r/debian • u/princepii • 2d ago
Nautilus says files are copied before they’re actually written to the USB drive
dell xps 15 7590 - Kernel Linux 6.12.113+deb13-amd64 - FW. 1.9.1 - gome48 - wayland
Today i tried copy a file to a usb stick. The transfer seemed unusually fast...it took about 20 seconds, which didn’t seem possible for a file of that size. Nautilus then said the file had been copied.
When i tried to eject the stick, however, i got a message saying the device was still writing. couldn’t see any write activity or related process in the system monitor, so i removed it anyway.
checked the drive on another device and found that the file hadn’t been fully copied. tried other usb sticks and drives, but the same thing happened. also ran sudo apt update, sudo apt upgrade, and sudo apt autoremove, then restarted, but that didn’t help.
i tried copying it again in Nautilus and used a stopwatch this time. Nautilus said the copy was complete after about 20 seconds, but after i ejected the drive using the right-click menu, it took around six minutes for the “Device removed” notification to appear. That seems closer to the actual time needed to copy a file of that size.
Does anyone know what might be causing this? it only started happening today. is this a known bug, or is there something i can try?
11
u/ZombieCrow 2d ago
Type "sync" in the terminal, when it starts blinking means its done.
3
u/princepii 2d ago
you mean when it stops blinking and you see the next prompt?
3
u/ZombieCrow 1d ago
When you type sync you wont be able to type other stuff in the terminal until its complete, thats what i mean. Also thats normal for larger files so just use sync and sont trust the file manager progress bar.
8
u/DerAndi_DE 2d ago
This is caching of file system writes, something that is useful as long as the media underneath is either fixed or fast enough to write larger amounts in reasonable time like a SSD. USB drives are neither, they are removable and orders of magnitude slower than SSDs. So it can happen that the file system driver in the kernel caches several hundred megabytes of data that need to be written, yet signaling user space that the operation is complete.
You could mount removable media with the 'sync' option, which forces the file system driver to write immediately, signaling 'done' only when it's really done. Downside is that copying small files gets a huge slowdown since these usually benefit most from caching.
The FAT filesystem has a "flush" option which flushes writes from the cache more often. It is a good compromise between 'sync' and 'async'. Debian resp. GNOME should use that by default.
Unfortunately, other filesystems like exFAT or ext3/4 don't have that. Ext3/4 is relatively uncommon on removable drives, but exFAT is fairly common and shows exactly that behaviour.
I know no other workaround other than opening a terminal and calling 'sync' once the copy is (apparently) complete. It actually can take several minutes to return, but then you can be sure everything has been written.
3
u/hdkaoskd 1d ago
The "eject" function in the UI (unmount) is the workaround.
2
u/DerAndi_DE 1d ago
As OP described, behaviour of 'eject' is not straightforward. The drive icon disappears even though there is still data being written. There is a notification telling you not to remove yet, but that disappears after some time, too, even though writes are not finished.
-2
u/princepii 2d ago
but i always use exfat...well most of the time and never got that behaviour.
i don't understand what happend since yesterday, that this happend. i didn't install anything new.
1
u/Real-Collection-5686 1d ago
most likely you didn't ever do such big writes or spend some time before removing
33
u/dougs1965 2d ago
'I got a message saying the device was still writing" ... "I removed it anyway" ... "is there something I can try?"
Try waiting.
-2
u/princepii 2d ago
but this is the first time nautilus is doing that.
even yesterday i copyd something and when it was done i got the done msg.
i use debian for years. never happend to me that way.
all are saying this normal. this that. that is thiswhy.
why was it not like that all the years i am using debian with gnome?
that is my question. i mean clearly something must have happend from yesterday to today.
it's not just the billo chinese usb stick. it's also samsung t3 and t5 ssd. also intenso and kingston usb stick.
i just wanna know if it is indeed normal...why now? it was never like that.
14
u/dougs1965 2d ago
It is normal caching as all the other comments say. I don't know why you weren't getting this behaviour previously, I don't know the specifics of your system, but when your system says the device is still writing, you should believe it.
-1
u/princepii 2d ago
so this should be ok? i mean the correct behaviour should be getting the final "files are copied" message when the files are actually copied and the writing process is done. not like after 20 seconds and it's actually still writing.
the ressource monitor not even showing writing process like before.
i don't get why this is all ok with everyone.
for me it's a big problem cus i wanna know that something is still copying without having to remove the drive to see it actually done. i don't want that after 20 seconds msg and it actually still copying...i want to see that the process is still copying and see it in the ressource monitor where there you can see what process is using cpu/gpu or what is still writing or reading.
9
3
3
1
u/arf20__ 1d ago
this is how fundamentally writing works in the linux virtual filesystem. you can know how much there is to write looking at the Dirty line of /proc/meminfo, and you can use the
synccommand to tell it to flush all the write cache to the drives. maybe it "didn't happen" (percepcionally) before because you didn't write big enough files to slow enough media. USB flash drives are very, VERY slow, regardless of the USB 3.2 whatever interface or how much DRAM cache they have3
u/abotelho-cbn 2d ago
It's absolutely not the first time Nautilus is doing that.
1
u/princepii 2d ago
em not a native english speaker and used translator to write the post. just so you know my english is not that good.
but can you explain what you mean?
you say that it always was like that and i just noticed it the first time?
bc i can assure you and i am 100% certain that in my system it happend the first time.
bc i always open the little process on the bottom left to see how fast it's writing and when it's done and it always was fine bc i move files a lot.
and thats why i didn't understand what was going on and that was also the reason why i tried to ask for help here.
and maybe i can not explain it the right way and that is maybe the misunderstanding here.
i deal a lot with different devices. like android, ios from family, windows from work and friends or collegues and also linux.
i am on debian for years. i also used other distros here and there for a while but always stuck with debian.
it always was just fine. copy file paste whereever...little popup windows that shows the process until it's done.
that changed today. i think it makes 0 sense to write it all over again what the problem is bc i really think ppl can understand what is written in the post.
3
u/abotelho-cbn 2d ago
It didn't change today.
It's possible you wrote larger files than normal. It has been this way forever.
2
2
u/rebootyourbrainstem 1d ago edited 1d ago
It's a known problem with Linux, if you have lots of memory Linux likes to keep a long queue of data to be written when it has time. This makes it a bit hard for applications, they can ask for data to be actually flushed to the drive, but doing at the end means they can't show real progress, and doing it all the time slows everything down.
So as for why now, maybe you didn't have a lot of free memory before, or maybe you just never wrote this much data at once, or maybe you never copied this much from a really fast disk to a really slow USB stick. Btw USB sticks can change speed a lot when they are older or nearly full. Basically impossible to know for us.
Anyway now you know that message is absolutely serious and you should NOT ignore it.
6
u/tombolda 2d ago
This also bugs me a lot on my system, but it's actually caching what you write to RAM but saying it's complete almost immediately. You can see this using eg Mission Center. It's marked as "Modified" (data that needs to be written to disk) Example, take my cheap chinese pendrive, sub 1MiB/s write speed, 2gb file copies over immediately, but in the background it'll keep copying it for 10 more minutes. Pressing eject during this will bring up the "Writing data to disk don't unplug yet" notification
6
u/michaelpaoli 2d ago
Yes, write(2) and close(2) can successfully return before all the data has completed being successfully written to media. This is for efficiency/performance reasons. sync(2) causes pending writes to media to be flushed to media. It may return before it completes that operation, however a second sync(2) can't start until an earlier pending sync(2) has completed. Likewise applies to related system calls.
So, e.g., you write a flash stick with dd(1), it may return before it completes writing the medial
But if after that dd completes, one does:
$ sync && sync
That won't return and complete without errors until at least that first sync(1) has successfully completed.
So, you want to be sure it's flushed out, follow it with
$ sync && sync
Nautilus ain't gonna dive that deep for you.
Also media such as USB flash sticks can be quite flakey - may fail at any time, and with zero warning. Even brand new good quality name brand from reputable supplier (so not fake flash, which is another issue) can be fault - even brand new. So best, do the write, when that's done, do the
$ sync && sync
And when that completes, remove the media, wait a wee bit (e.g. 10 seconds), reinsert it, and then only after that, read the entire media (or at least the data of interest, e.g. to the length of the ISO image), and compare it to the source data it was copied from and should match to (or compute secure hash and check that). Yeah, not too many years back, was helping someone install Linux on a laptop at a Linux User Group meeting. It kept failing part way through the install. Compared the ISO data on flash stick to that of the verified ISO file - didn't match. So, wrote it again, doing the sync && sync and all that - no errors, but after removing it, waiting a bit, reinserting it, and then comparing - it failed to match - it was silently corrupting data on write, giving no errors when doing so, and could only see that when reading it all back to check. Also, the reason to remove and reinsert, is so one fully reads data from that device, not reading it (or partially so) from cache in RAM. Anyway, that was a brand new flash stick from good manufacturer and reputable supplier ... it came packaged as a 2-pack. Then tried the other one that came out of that same pack - no problems at all, worked perfectly fine.
Likewise for optical. Don't presume it wrote the data okay just because there were no write errors or such. Remove the media, wait wee bit, reinsert, then read it back and compare or do secure hash to verify. Yeah, Nautilus ain't never gonna be that smart and thorough about such operations.
And these caching effects can be hugely noticeable, e.g. system with lots of RAM, and writing very large image/file to USB flash - the write will return long before it completes writing the data to media. And that's mostly a very good thing. Think of your highly busy application that's super frequently overwriting same data in a file or device. Without such caching that would slow to the speed of those media writes - way the hell slower than RAM. But with caching it doesn't have to write all that out - only periodically flush out to media the net writes that are still pending. Hute performance boost, also majorly cuts down on media wear (be it solid state / flash, or HDD).
3
u/jr735 Debian Testing 1d ago
I found out, experimentally, that one sync is sufficient. I always believed the double sync, but it's not necessary, it would seem.
3
u/michaelpaoli 1d ago
Yeah, I always do two - works on any *nix, don't have to so much as give it a second thought.
But if you're never on a *nix that isn't Debian (or non-ancient Linux), that may not be a concern.
Likewise:
$ sudo su -
or just
$ su -
if sudo isn't there. Works fine and consistently on any *nix, even going back long before Debian or Linux existed. None of that question of what options and syntax exactly to properly initialize the environment for the target user and not get any unpleasant surprises. And for commands where that is or may be relevant,
$ [sudo] su - user -c 'command [args] ...'
4
u/willyhun 2d ago
Does anyone know what might be causing this? it only started happening today. is this a known bug, or is there something i can try?
No, this is not a bug. Nautilus does not know what is happening with your files, it issues the file operation and got a confirmation from the fs subsystem.
That's not Debian, not even Linux, it is the same on all OS.. You have to wait until you can safely remove the device.
2
u/TomDuhamel 1d ago
Akschtually...
It is Linux. What else did you think it was?
Windows disabled disk cache on removable drives because of users being confused and constantly pulling them out before a write has finished.
1
0
1d ago
[removed] — view removed comment
-1
u/debian-ModTeam 1d ago
This post has been removed as it was either reported to and/or acted upon by mods to be found in violation of Rule #1 regarding not being in line with expected discourse etiquette or the Debian Code of Conduct.
1
u/hdkaoskd 1d ago
It is Linux. Operating systems have write caches.
1
u/neoh4x0r 1d ago edited 1d ago
It is Linux. Operating systems have write caches.
Yes write caches are a thing, but it's an attempt to solve a common problem inherent in systems design rather than being an OS-specific issue.
In other words, it's not Linux, it's operating systems in general, and more to the point it's actually an architectural hardware issue.
What it really means is that if write caches are actually a problem then fundamental systems design should incorporate solutions that do not need these types of buffers.
0
u/willyhun 1d ago
Yep, all of them, you apparently don't know Linux because you don't know what a sync mound does.
3
u/psyblade42 1d ago
I use a udev rule to limit the write cache of removable storage such as usb sticks. This keeps the overall speed improvement of cache but with a much shorter time to remove.
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{removable}=="1", ATTR{bdi/max_ratio}="1"
bdi/max_ratio is the percentage of ram. This defaults to 20%.
8
u/PowerBIEnjoyer 1d ago
I love how everyone just blindly downvotes comments when this is an obvious UX issue of Linux based operating systems. Really similar behavior to how Arch users behave towards users wanting to use AUR.
Behaving like that will surely keep Linux growing I am sure.
4
u/foxtrotgulf 2d ago
It's called a write cache which helps to improve the performance of your storage. This is expected behavior and is happening down in the kernel. From Nautilus' perspective, the file has been written and so it reports it as so. You should wait until the device fully unmounts before removing it.
2
u/Euroblitz 2d ago
That's normal on linux, not exclusive from Nautilus. Remove drive safely and wait until it finishes removing
-2
u/Bulky-Hair8606 2d ago
It is sorta exclusive to nautilus. Dolphin auto syncs. Same with thunar, it also auto syncs.
4
2
u/Dolapevich 2d ago edited 1d ago
So, this is the expected behaviour. What you see is called "write cache", the idea being that when something is expected to write in the disk, which is a really slow operation comparatively, it will not wait for the write to complete before returning.
This is a very usefull strategy on slow disks. The kernel is smart enough to show the file where it is should be, even when it is still not there.
On removable devices sometimes it backfires, as you can see. That is the necesity of the "safely eject" button, whose role is flush everything to the device before umounting.
However, if the write cache was too agressive, and ram is available, it might be the case that the whole file is cached in ram, and slowly writen to the device, so when you issue the eject, it will keep writing until it can flush everything.
There are two alternatives:
- Limit the size of ram available for this, so write will stay on it until the last part, return. And when eject is used just a bit of the file will need to be flushed.
- Disable write cache altogether.
This is a user patience/knowledge issue and a performance technique. Windows does disable write cache in removable devices, and that is the reason it is almost never an issue.
If you'd like to limit the available ram to the USB write cache, follow this.
It will instruct udev to run a low level configuration to limit how much ram the device will have available.
1
u/DeepSea_Dreamer 22h ago
I'm more surprised it started happening now for you. This is normal behavior - the system loads it into cache and copies it when it has time. Usually, people don't need to dismount their partition right after they finish copying, and when they do, that's when they click eject (or however it's called) and wait for the cache to be flushed.
It has no downside - the time you have to wait for the flash drive to be unmounted will never longer than if you wrote the files synchronously.
1
u/neon_overload 2d ago
There's a lot of precedent on computers that you don't pull out a removable drive without unmounting ("eject" in GUI) as recent writes may not have finalised.
Some systems will only enable write caching for certain filesystems - for example, FAT32 and exFAT sometimes will be mounted without write caching (all writes are "sync"). So, if you're used to this not being a problem on other drives, maybe it was because those were FAT32 or exFAT while this is something more typically used on a permanent drive or where more performance may be desired.
1
1
u/MCBuilder30140 1d ago
so I'm not the only one!
for me it the time it took to copy seemed fine to me but it refused to eject right after
I see that you have a Ventoy drive attached and I was too copying stuff to a Ventoy USB stick
maybe it's Ventoy because I don't remember having this issue on other drives on my Ubuntu laptop...
2
23
u/charles25565 2d ago
This is because Linux caches writes in RAM heavily as dirty pages.
You can check how much data needs to be written using
grep Dirty /proc/meminfo. You can prefix it withwatch -n 0.1to get a live status.The
synccommand will forcibly flush and block until it is flushed.