Noufal Ibrahim
constrained creativity

Very often during my work, I’ve needed to treat a list of things (usually stored in a file one per line) as a set. This means that we should be able to to find the union of two files, intersection of two files etc. For example, if one file A looks like this

item1
item2
item3
item4

and file B looks like

item3
item4
item5
item6

Their union should be a file that has item1, item2, item3, item4, item5 and item6. The intersection would be a file that has item3 and item4. A - B would be the set of all elements in A which are not in B (in this case item1 and item2). The symmetric difference of A and B would be the set of elements in A or B but not in both. In other words (item1, item2, item5 and item6). It’s possible to script my way through this tangle with the usual UNIX tools and a bunch of temporary files but I needed these often enough to justify a new tool.

This was born the unimaginatively named lines (PyPI link). It gives you a command line program to do things like this. Here are a few examples.

(x)% cat examples/f1
a
b
c
d
(x)% cat examples/f2
c
d
e
f
(x)% python lines.py -u examples/f1 examples/f2 #Union
a
c
b
e
d
f
(x)% python lines.py -i examples/f1 examples/f2 # Intersection
c
d
(x)% python lines.py -d examples/f1 examples/f2 # difference (f1 - f2)
a
b
(x)% python lines.py -s examples/f1 examples/f2 # symmetric difference (xor)
a
b
e
f
(x)%

I’ve found this useful when I need to deal with lists of file names or lists of tests.

I added two more features which are more flaky but still useful. The first one squeezes empty lines from a file.

(x)% cat examples/f3
a
b

c

d

e
f
(x)% python lines.py --squeeze examples/f3
a
b
c
d
e
f
(x)%

The other is more interesting which I first wrote while working for a an Internet Archive project. With a very long set of elements that are almost similar, it’s useful to be able to figure out anomalies. In my case, I had a list of item name. Each item name looked something like this cuil-domainshard-corpus5-large-merge-rev1.00478-of-25000 The 00478 would change. There would be a few items in this list that didn’t “fit the pattern” usually caused by an interrupted transfer or something similar. It’s useful to be able to weed these out quickly. The --patterns option looks through the entire list and partitions the set into subsets. The elements of a subset will all have an upper bound on their Levenshtein distance from one another. This means that for a file like this

Archive.001-of-020.part
Archive.002-of-020.part
Archive.003-of-020.part
Archive.004-of-020.part
Archive.005-of-020.part
Archive.006-of-020.part
Archive.007-of-020.part
.Archive.008-of-020.part.zbnrw
Archive.009-of-020.part
Archive.010-of-020.part
Archive.011-of-020.part
Archive.012-of-020.part
Archive.013-of-020.part
Archive.014-of-020.part
Archive.015-of-020.part
Archive.016-of-020.part
Archive.017-of-020.part
Archive.018-of-020.part
Archive.019-of-020.part
Archive.020-of-020.part

Running python lines.py --patterns examples/f6 -l 5, would give me

19 elements
1 elements - {'.Archive.008-of-020.part.zbnrw'}

This means that there are 19 elements in one subset and 1 in another. The -l option allows us to tune the upper bound on the levenshtein distance and the -p parameter allows us to configure a “outlier percentage”. If the number of elements in a subset is less than this percentage, the elements will be printed out in the output.

This is an alpha release so it’s probably buggy. Comments and feedback are welcome.