The tar archive format, its extensions, ...

P'skhal · Le 30/07/2022, à 09:41

... and why GNU tar extracts in quadratic time

A couple of days ago, I had a 518GiB tar.gz file (1.1 TiB uncompressed) that I had to extract. At first, GNU tar was doing a great job, chewing through the tar.gz at around 100MiB/s. But after a while, it slowed significantly; down to less than a kilobyte per second. pv's time estimate went from a bit over an hour, to multiple hours, to over a day, to almost a week. After giving it some time, and after failing to find anything helpful through Google, I decided that learning the tar file format and making my own tar extractor would probably be faster than waiting for tar. And I was right; before the day was over, I had a working tar extractor, and I had successfully extracted my 1.1TiB tarball.
I will explain why GNU tar is so slow later in this post, but first, let's take a look at:
The original tar file format
Tar is pretty unusual for an archive file format. There's no archive header, no index of files to fascilitate seeking, no magic bytes to help file and its ilk detect whether a file is a tar archive, no footer, no archive-wide metadata. The only kind of thing in a tar file is a file object.

La suite : https://mort.coffee/home/tar/

M'lou · Le 30/07/2022, à 10:02

In French please ? tongue

J'ai quand même compris de quoi ça parle

P'skhal · Le 30/07/2022, à 22:03

big_smile

Ouaip, le gars s'est concocté un tar beaucoup plus efficace vis à vis de son cas mais bon, ce n'est pas très courant d'extraire 1 To de données je pense... tongue

M'lou · Le 31/07/2022, à 10:06

Ah ? tongue lol

P'skhal · Le 31/07/2022, à 10:17

lol lol lol

Bon, je retire ! r_tirelangue7

M'lou · Le 31/07/2022, à 10:22

lol lol lol lol wink

P'skhal · Le 31/07/2022, à 10:23

wink

P'skhal	#1 Le 30/07/2022, à 09:41
Explorateur Lieu : Au-delà du portail Inscription : Le 18/08/2016, Messages : 37 373	... and why GNU tar extracts in quadratic time A couple of days ago, I had a 518GiB tar.gz file (1.1 TiB uncompressed) that I had to extract. At first, GNU tar was doing a great job, chewing through the tar.gz at around 100MiB/s. But after a while, it slowed significantly; down to less than a kilobyte per second. pv's time estimate went from a bit over an hour, to multiple hours, to over a day, to almost a week. After giving it some time, and after failing to find anything helpful through Google, I decided that learning the tar file format and making my own tar extractor would probably be faster than waiting for tar. And I was right; before the day was over, I had a working tar extractor, and I had successfully extracted my 1.1TiB tarball. I will explain why GNU tar is so slow later in this post, but first, let's take a look at: The original tar file format Tar is pretty unusual for an archive file format. There's no archive header, no index of files to fascilitate seeking, no magic bytes to help file and its ilk detect whether a file is a tar archive, no footer, no archive-wide metadata. The only kind of thing in a tar file is a file object. La suite : https://mort.coffee/home/tar/ “ Aujourd'hui plus qu'hier et bien moins que demain „
Hors ligne

M'lou	#2 Le 30/07/2022, à 10:02
MétalWoman Lieu : Quelque part dans la Galaxie Inscription : Le 18/08/2016, Messages : 41 725	In French please ? J'ai quand même compris de quoi ça parle L'amour est comme un jardin, il doit se cultiver chaque jour pour donner le meilleur de lui-même
Hors ligne

P'skhal	#3 Le 30/07/2022, à 22:03
Explorateur Lieu : Au-delà du portail Inscription : Le 18/08/2016, Messages : 37 373	Ouaip, le gars s'est concocté un tar beaucoup plus efficace vis à vis de son cas mais bon, ce n'est pas très courant d'extraire 1 To de données je pense... “ Aujourd'hui plus qu'hier et bien moins que demain „
Hors ligne

M'lou	#4 Le 31/07/2022, à 10:06
MétalWoman Lieu : Quelque part dans la Galaxie Inscription : Le 18/08/2016, Messages : 41 725	Ah ? L'amour est comme un jardin, il doit se cultiver chaque jour pour donner le meilleur de lui-même
Hors ligne

P'skhal	#5 Le 31/07/2022, à 10:17
Explorateur Lieu : Au-delà du portail Inscription : Le 18/08/2016, Messages : 37 373	Bon, je retire ! “ Aujourd'hui plus qu'hier et bien moins que demain „
Hors ligne

#1 Le 30/07/2022, à 09:41

The tar archive format, its extensions, ...

... and why GNU tar extracts in quadratic time

The original tar file format

#1 Le 30/07/2022, à 09:41

... and why GNU tar extracts in quadratic time

The original tar file format

#2 Le 30/07/2022, à 10:02

Re : The tar archive format, its extensions, ...

#2 Le 30/07/2022, à 10:02

#3 Le 30/07/2022, à 22:03

Re : The tar archive format, its extensions, ...

#3 Le 30/07/2022, à 22:03

#4 Le 31/07/2022, à 10:06

Re : The tar archive format, its extensions, ...

#4 Le 31/07/2022, à 10:06

#5 Le 31/07/2022, à 10:17

Re : The tar archive format, its extensions, ...

#5 Le 31/07/2022, à 10:17

#6 Le 31/07/2022, à 10:22

Re : The tar archive format, its extensions, ...

#6 Le 31/07/2022, à 10:22

#7 Le 31/07/2022, à 10:23

Re : The tar archive format, its extensions, ...

#7 Le 31/07/2022, à 10:23

Pied de page des forums