The Challenges of Cross-App Copy-Paste and Our Solutions

Published:
Cover.png

Craft is a powerful note-taking and doc creation app.

Designed for those who love simple but powerful productivity tools. Write notes quickly and make them beautiful with AI in seconds. Get things done beyond docs with whiteboards, tasks, and reminders securely synced across all your devices.

You can apply commonly used styling, such as making a text look like a title, heading, or strong, changing its font type, ordering it in bullet or numbered lists, and adding some inline styling: making a range in text bold or italic.

Craft can also support advanced structures like toggles, focus/quote blocks, and tables.

But Craft shines when you start structuring your thoughts into pages and cards. Make them expressive by setting backgrounds, cover images, and page breaks.

image

Craft's basic unit of building is a block. It can be a word, a sentence, or a paragraph. You can drag, move, copy, style, convert it into pages and cards to structure, and make your document more excellent. This is a newer approach to text editing (also used by Notion) compared to existing document editors. They usually focus on longer, continuous texts.

When you have created beautiful documents in Craft, you may want to continue your work in a different application and move your content over to it. One way to export your content is to copy from Craft and paste it into your target application.

This raises the first question:

How do we make content created in Craft compatible with other applications?

And if that is solved, we still have a second question:

How can we ensure that this content is moved to these applications correctly?

The simple user interaction of copy pasting, hides complex logic, edge cases, and fighting with the system under the hood. I will give context and explain how we solve these problems in Craft.

Craft's earliest supported platforms were iOS and Mac(via macCatalyst), and I'm an iOS/Mac product engineer so that I would focus on these platforms. I'm sure our web and Windows developers could tell similar stories.

The problem of transferring the content of the document between different applications is complex. To help understand it, I will use the analogy of being at a party and talking to others in different languages.


image

Imagine you are invited to a party

This party is held on your Apple device. You heard that some of the more famous guests (Notion, Microsoft Word, Slack, Bear, Obsidian) are there, and you, being the new guy in the block (only four years old), can't wait to talk with them and share your knowledge and how you see the world.

You are familiar with the basics. If you want to share your content with these apps via copy/paste, you need to access the UIPasteBoard by registering your format of your content.

class BlockNSItemProvider: NSItemProvider {
  init(){
    registerDataRepresentation(forTypeIdentifier: kUTTypeUTF8PlainText as String, visibility: .all) { (completionBlock) -> Progress? in
      let progress:Progress = Progress(totalUnitCount: 0.0)
      //Convert the block data to String, then to UTF-8 data, also handle errors
      let data:Data = ...
      progress.completedUnitCount = 100
      completionBlock(data, nil)
      return progress
  }
}

Example how Craft registers its blocks as a plain UTF-8 text to the pasteboard, for others to use it.


image

Imagine the guests are speaking multiple languages

The story gets trickier as you learn that you need to express yourself in different forms to make sure your ideas are heard by the other guests. That means that Craft needs to translate its blocks into multiple languages, sometimes even multiple flavors:

  • a simple text block should be available in plain text (for simple editors), HTML for browsers, rich text format for Apple Notes and rich editors, markdown for most of the other editors (also Slack).
  • an image, video, drawing should be available as a multimedia content if the target application is supporting it, or as an URL in a form that will be convenient
  • other structuring items like pages, cards, toggles can't be represented in linear editors (Bear, Words), so they are flatten out either nested to their parent content, or appended after the parent's content.

Each format has its own level of expression and limitation. For each block that is selected, we convert the internal representation in all popular UTType format:

plain text: we just copy the plain String content of the block, get its UTF-8 and UTF-16 representation as data

markdown: we use a markdown parser to convert the block with its internal range attributes into a markdown text, then the string is registered as plain text.

HTML/RTF: we convert the block into an NSAttributedString and use it's document-related APIs to convert them into HTML or RTF format. I will describe our process through the HTML conversion (same is true for RTF):

let blockAttributedString:NSAttributedString = ... //Created from internal block representation
let htmlData:Data = try self.data(from: NSRange(location: 0, length: self.length), documentAttributes:[.documentType: NSAttributedString.DocumentType.html]);
let htmlString:String = String.init(data: htmlData, encoding: String.Encoding.utf8)

We are using the built in Apple API here. We only use the RTF and HTML from the supported types.

If we take this example sentence: This answers our first question.

This answers our {
    NSColor = "UIExtendedSRGBColorSpace 0.121569 0.133333 0.145098 1";
    NSFont = "\".SFNS-Regular 15.00 pt. P [] (0x11592a450) fobj=0x11592a450, spc=3.98\"";
    NSParagraphStyle = "Alignment Natural, LineSpacing 0, ParagraphSpacing 0, ParagraphSpacingBefore 0, HeadIndent 0, TailIndent 0, FirstLineHeadIndent 0, LineHeight 22.5/0, LineHeightMultiple 1.25, LineBreakMode WordWrapping, Tabs (\n    28L,\n    56L,\n    84L,\n    112L,\n    140L,\n    168L,\n    196L,\n    224L,\n    252L,\n    280L,\n    308L,\n    336L\n), DefaultTabInterval 0, Blocks (\n), Lists (\n), BaseWritingDirection Natural, HyphenationFactor 0, TighteningForTruncation NO, HeaderLevel 0 LineBreakStrategy 0 PresentationIntents (\n) ListIntentOrdinal 0 CodeBlockIntentLanguageHint ''";
}first{
    NSColor = "UIExtendedSRGBColorSpace 0.0784314 0.0862745 0.0941176 0.945098";
    NSFont = "\".SFNS-Bold 15.00 pt. P [] (0x11b333990) fobj=0x11b333990, spc=3.63\"";
    NSParagraphStyle = "Alignment Natural, LineSpacing 0, ParagraphSpacing 0, ParagraphSpacingBefore 0, HeadIndent 0, TailIndent 0, FirstLineHeadIndent 0, LineHeight 22.5/0, LineHeightMultiple 1.25, LineBreakMode WordWrapping, Tabs (\n    28L,\n    56L,\n    84L,\n    112L,\n    140L,\n    168L,\n    196L,\n    224L,\n    252L,\n    280L,\n    308L,\n    336L\n), DefaultTabInterval 0, Blocks (\n), Lists (\n), BaseWritingDirection Natural, HyphenationFactor 0, TighteningForTruncation NO, HeaderLevel 0 LineBreakStrategy 0 PresentationIntents (\n) ListIntentOrdinal 0 CodeBlockIntentLanguageHint ''";
    ourSpecialBoldFlag = 1;
} question.{
    NSColor = "UIExtendedSRGBColorSpace 0.121569 0.133333 0.145098 1";
    NSFont = "\".SFNS-Regular 15.00 pt. P [] (0x11592a450) fobj=0x11592a450, spc=3.98\"";
    NSParagraphStyle = "Alignment Natural, LineSpacing 0, ParagraphSpacing 0, ParagraphSpacingBefore 0, HeadIndent 0, TailIndent 0, FirstLineHeadIndent 0, LineHeight 22.5/0, LineHeightMultiple 1.25, LineBreakMode WordWrapping, Tabs (\n    28L,\n    56L,\n    84L,\n    112L,\n    140L,\n    168L,\n    196L,\n    224L,\n    252L,\n    280L,\n    308L,\n    336L\n), DefaultTabInterval 0, Blocks (\n), Lists (\n), BaseWritingDirection Natural, HyphenationFactor 0, TighteningForTruncation NO, HeaderLevel 0 LineBreakStrategy 0 PresentationIntents (\n) ListIntentOrdinal 0 CodeBlockIntentLanguageHint ''";
}
{
    NSFont = "\".SFNS-Regular 16.00 pt. P [] (0x11b344de0) fobj=0x11b344de0, spc=4.19\"";
}

It will be converted into the following HTML:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="2487.5">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; line-height: 22.5px; font: 15.0px \'.AppleSystemUIFont\'; color: #1f2225}
span.s1 {font-family: \'.SFNS-Regular\'; font-weight: normal; font-style: normal; font-size: 15.00px}
span.s2 {font-family: \'.SFNS-Bold\'; font-weight: bold; font-style: normal; font-size: 15.00px; color: rgba(20, 22, 24, 0.95)}
</style>
</head>
<body>
<p class="p1"><span class="s1">This answers our </span><span class="s2">first</span><span class="s1"> question.</span></p>
</body>
</html>

You can see that it generates a full HTML page, with html, head and body tags. The attributes are converted into CSS styles and referenced later in the body part via class. Each class previously was one distinct attributed range in the original attributed string.

  • image/video: we download or use the cached image/video and put that on the pasteboard with the appropriate UTType type (for image we use public.png)
  • smart and normal urls: are added as plain and public.url types.

When you select a simple text block and hit copy, this is the state of the clipboard (made visible via a Clipboard viewer app).

image

This answers our first question: how to transform the Craft-world specific blocks into standard paste board formats that others can read.

The devil is in the details; how you put data on the pasteboard is really important. Let's continue with our party analogy.

image

Imagine you speak some foreign languages really well, but some of them you ignored in school

At Craft, first we try to focus on the most important/critical features that gives value for the users, then later we improve on them after feedbacks and see what is really important for the user. This was the case with Copy/Paste too. This is a must-have essential feature in every document editor, so it was added at the very beginning of the very first release of Craft in November 2020. We first focused on supporting plain text and markdown right, back at the time that seemed important.

After all, markdown and plain text are the English language of document editors, everybody should support it, right?

Except that the world and also Craft is changing. Other apps added and started to prefer and support HTML or RTF.

Meanwhile, Craft also introduced even more complex structure like table, that were not integrated into the pasteboard support correctly.

These two resulted in the scenario: If you would happen to copy a table and paste it into an application that uses HTML, you would only see "Table" (literally) pasted.

We started to receive feedback to improve our HTML skills, and also recommended popular apps.

"[…] I LOVE craft. it's the first app that I've actually stuck with — and I've tried Notion, Roam, Evernote, etc. There's one feature, however, that my entire team is struggling with: copy/paste. I've attached a screenshot of some text on Craft —and how it looks when pasted into Google Docs. The main issue is around spacing with bullets — the formatting doesn't get preserved. […]"

"[…] However, my team also uses email, Slack, and Google Docs for other means of communication. & collaboration This often requires me to copy/paste my notes into these other apps. Unfortunately, the styling/formatting doesn't transfer 1:1 — and I need to constantly delete tabs, etc. […]"

Some of the recommendations were web based (Hey, Google Documents, Gmail web). On web the Clipboard API gives support mainly for plain text and html format. So we decided to be better in HTML.

In Craft version 2.7.9 we introduced support for better HTML for inline styling, list items and tables.

How did we improve HTML conversion for inline styling?

The original problem was that even you use platform native APIs to create the HTML string, other might come from a cross platform environment, and they will simply ignore parts of the HTML string. This caused users loosing formatting during copy paste. No one can blame them, malicious apps could hide runnable javascript code into the <head> part, and they do not want to execute them on parsing. So they strip the head part and only use the content of the body.

To overcome this, we made all style inline, a bit ugly, but simple solution. So the above example looks like this:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="2487.5">
<style type="text/css"></style>
</head>
<body>
<p style="font:15.0px '.AppleSystemUIFont';color:#1f2225;margin:0.0px 0.0px 0.0px 0.0px;line-height:22.5px"><span style="font-style:normal;font-weight:normal;font-size:15.00px;font-family:'.SFNS-Regular'">This answers our </span><span style="font-family:'.SFNS-Bold';font-size:15.00px;font-style:normal;color:rgba(20, 22, 24, 0.95);font-weight:bold">first</span><span style="font-style:normal;font-weight:normal;font-size:15.00px;font-family:'.SFNS-Regular'"> question.</span></p>
</body>
</html>
</div>

This helped improve preserving formats for applications like Hey or Gmail on web, when user used Copy as HTML.

How did we improve HTML conversion for list items?

Craft supports toggles, numbered, and bullet lists. Previously, they were all converted to RTF/HTML format by adding a dash before the block content. Users would expect proper, alternating list prefix representation, which you can see in the Craft editor itself. We solved this issue by adding NSTextList items to the paragraph attribute of the block-attributed string.

You need to add a decimal text list item to have numbered lists: NSTextList(markerFormat: .decimal, options: 0) and you can add disc/circle items for bullet lists NSTextList(markerFormat: nested_level%2 == 0 ? .disc : .circle, options: 0)

let item:NSTextList = NSTextList(markerFormat: .decimal|.disc|.circle, options: 0)
let p:NSMutableParagraphStyle = NSMutableParagraphStyle()
p.textLists = [item]

blockAttributedString.addAttributes([.paragraphStyle : p])

Putting our example into a first-level bullet list would change the converted attributed string to

This is the answer for the {
    NSColor = "UIExtendedSRGBColorSpace 0.121569 0.133333 0.145098 1";
    NSFont = "\".SFNS-Regular 15.00 pt. P [] (0x12d166c90) fobj=0x12d166c90, spc=3.98\"";
    NSParagraphStyle = "..., Lists (\n    \"NSTextList 0x6000077e54a0 format <{circle}>\"\n)...";
}first{
    NSColor = "UIExtendedSRGBColorSpace 0.0784314 0.0862745 0.0941176 0.945098";
    NSFont = "\".SFNS-Bold 15.00 pt. P [] (0x36fc862c0) fobj=0x36fc862c0, spc=3.63\"";
    NSParagraphStyle = "..., Lists (\n    \"NSTextList 0x6000077e54a0 format <{circle}>\"\n)...";
    lukiBold = 1;
} question
{
    NSColor = "UIExtendedSRGBColorSpace 0.121569 0.133333 0.145098 1";
    NSFont = "\".SFNS-Regular 15.00 pt. P [] (0x12d166c90) fobj=0x12d166c90, spc=3.98\"";
    NSParagraphStyle = "..., Lists (\n    \"NSTextList 0x6000077e54a0 format <{circle}>\"\n)...";

Two important things to note here:

  1. if you decide to support nested lists with indentation, then make sure, that you are using the same NSTextList object when setting the attributes for a range. This helps NSAttributedString to unify the attributed ranges, and the conversion will create nice UL/LI list items.
  2. When you concat multiple lines with line breaks, make sure that the \n at then end is also part of the range, where the text list is applied to.

Here is an example result after adding another nested list to the block:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="2487.5">
<style type="text/css"></style>
</head>
<body>
<ul style="list-style-type:circle">
    <li style="font:15.0px '.AppleSystemUIFont';margin:0.0px 0.0px 0.0px 0.0px;color:#1f2225"><span style="font-size:15.00px;font-family:'.SFNS-Regular';font-weight:normal;font-style:normal">This is the answer for the </span><span style="font-size:15.00px;font-style:normal;font-weight:bold;color:rgba(20, 22, 24, 0.95);font-family:'.SFNS-Bold'">first</span><span style="font-size:15.00px;font-family:'.SFNS-Regular';font-weight:normal;font-style:normal"> question</span>
    </li>
    <ul style="list-style-type:disc">
        <li style="font:15.0px '.AppleSystemUIFont';margin:0.0px 0.0px 0.0px 0.0px;color:#1f2225"><span style="font-size:15.00px;font-family:'.SFNS-Regular';font-weight:normal;font-style:normal">This is the answer for the </span><span style="font-size:15.00px;font-style:normal;font-weight:bold;color:rgba(20, 22, 24, 0.95);font-family:'.SFNS-Bold'">second</span><span style="font-size:15.00px;font-family:'.SFNS-Regular';font-weight:normal;font-style:normal"> question</span>
        </li>
    </ul>
</ul>
</body>
</html>
</div>

Web developers might already see one problem with the Apple converter: It is not generating standard nested list syntax! The second level <ul> tag should start inside the first level's <li> tag. (See w3school example). Some applications are fine with this; some, like Gmail on iOS, is not, resulting in extra bullet points at the start of each new nested list.

How to improve HTML conversion for tables?

The Apple built in API can't create tables, because NSAttributedString on iOS is not supporting it. (They do on macOS with the NSTextTable objects in the NSParagraphStyle). We solved this by converting our custom table representation into a HTML table format. To preserve the table tags during NSAttributedString HTML conversion, we create the HTML string for the tables, but insert a unique tag into the string, after the conversion, we replace the tokens by the table content. We even added support for the same inline format conversion for individual cells that we solved above for normal text blocks.

image
Instead of the word "Table", the user now has a nice table that supports text alignment, (background) colors, formulas, emojis etc.

Awesome, Craft finished the HTML language course, we are in the clear, right?… right... RIGHT?

image

Even though you speak popular languages well, some apps might prefer your worst language.

Let's turn off the lights at the party. We can hear others shouting in different languages but not know who spoke; we only recognize the various languages. On the Apple platform, it is up to the receiver to decide in which order and how they will listen to what they hear.

You can speak your best HTML, if the receiver application is looking for RTF first.

The receiver can guess the source application for a copy operation, because applications are putting some extra content to the pasteboard, that will reveal something of their identity:

  • Apple iWork applications like pages, numbers will put extra iWork-marked content (e.g. com.apple.iWork.TSPNativeData)
  • Adobe will put an extra com.adobe.pdf
  • Safari will put an extra com.apple.webarchive

Good guy apps, might use some heuristic or knowledge to change how and in which order they will scan the UIPasteboard to provide a better collaboration.

We try to be nice and listen to these special voices during pasting, adding special cases to handle imports from other popular apps better. Other apps might not, and the result is a poor user experience in the end.

We also tried to change the game and gain some control over the way in which a receiver is using our pasteboard data, by allowing the user to decide in which format they want to have their data. We support single plain text/markdown/HTML/RTF copy, where we only put the data in the selected format on the pasteboard. This will force the receiver to focus on the language we speak better (or not hear us at all, if it turns out that it is not supporting the format).

Even when everything is right, there are still other factors that can cause issues:

  • A new OS version can break putting different content of on the pasteboard: In iOS 17 the URL(string:) API changed, no longer returns nil, if the provided string is not an URL. Possibly this resulted in the RayCast app bug, that puts an URL content on the pasteboard even when the copied data is a simple text. Craft tried to paste this as an escaped URL content into a block.
  • A new receiver app version can change the behavior how your data will be parsed, also can have different implementation by platforms (Gmail on web can parse Apple HTML content well, but the Gmail iOS application has troubles with nested NSTextLists)

image

Imagine that your voice is shared through a man in the middle

The lights are down at the party, and you speak only one language to make sure others hear you better, but everything you say is actually heard and transmitted by another person.

This is the case with our Mac application, which uses the macCatalyst framework. This framework helped us bootstrap our top iPad code and has good macOS support. We learned a lot from this transition (see blog post here).

Once we started to play with Copy as HTML/RTF, we learned another interesting behavior of the framework. On iOS, when you register an NSItemProvider for the pasteboard, you define the format type and a load handler, that will gather the data asynchronously, when the receiver app decides to use that format. On Mac, the macCatalyst framework has to convert the asynchronous code to a synchronous AppKit API. This means that whenever you register a format, the load handler will be called instantly, blocking the main thread and inviting the spinning beach ball to this party.

Above this, the items put on the pasteboard is different, than the ones a native AppKit API would add by default. Let's compare the content of the paste boards in these two cases:

image
Here you see what the macCatalyst framework puts on the pasteboard when we register our data as RTF. (Minus the json one).
image
Here you see what the AppKit framework puts on the pasteboard when we register our the same data as RTF with the Appkit API.

 

This results in some native and Electron app (like Slack, VSCode) not recognizing HTML and RTF content correctly, even though the developer put them on the pasteboard in the correct format.

We are planning to change this part of the application to use the AppKit framework instead.

This is the answer for the second question: To help transferring the content the right way, we improved our support for specific languages, allowed the user to control how the data is copied on the clipboard and used even more native solution.

Closing thoughts

I hope this post showed the complexity of handling copy-pasting correctly. I tried to highlight the context behind our decision and the way our features are designed in recent releases.

If you want to improve the copy experience in your app, here are some takeaways:

  • Be aware what is the current state of the paste board. You can use any app, we used the free Clipboard Viewer. This can help you see what type of content is put on the pasteboard.
  • There are many formats, options to support, try to focus on improving the ones your users are missing or relying on every day (plain text, markdown, HTML and RTF for document editors are the most used).
  • Avoid doing long and CPU intensive work in NSItemProvider's registerDataRepresentation loadHandlers. On macCatalyst, you might freeze your application.
  • To control how other apps consume your content, provide "Copy as" functionality. This might be an advanced/pro feature, but it helps the user be a part of the decision process.
  • As you can't know how the receiver app will handle your data, convert it into standards, popular, and safe format (like focusing on the body one from an Apple converted HTML string)

The above story is only one side of the coin when you copy your content out of Craft. The pasting problem is just as hard as this. This time, you are the one in the dark room, trying to make sense in the multilingual screaming. In a future post, we will elaborate on that, too.

If you have any thoughts, idea, technical questions on this blog post or comments, please email me at peter@craft.do . Thank you for reading!

Interested? Read More...