Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How do I reliably get the kerning information into a file? #3811

Open
NathanWailes opened this issue Jul 21, 2019 · 28 comments
Open

How do I reliably get the kerning information into a file? #3811

NathanWailes opened this issue Jul 21, 2019 · 28 comments

Comments

@NathanWailes
Copy link

NathanWailes commented Jul 21, 2019

I am working off the code found here to try to extract kerning information from TTF fonts using fontforge.

I'm running the program against Google's entire set of free fonts.

Sometimes it works, sometimes it doesn't:

What was extracted:
image

What the FontForge UI shows:
image

How can I reliably get that "Kern" number that shows up in the UI, except in text form, so that I can reference it in a Python program?

Here's the code I'm using:

# run these two commands in the fontforge "embedded" python interpreter (ffpython.exe)
# >>> script = open( "C:\\Users\\<your-name>\\path\\to\\this\\file.py", "r" )
# >>> exec script
import json
import os

import fontforge

USER_NAME = 'Nathan'
PATH_TO_FONTS_FOLDER = 'C:\\Users\\Nathan\\Desktop\\fonts'
PATH_TO_OUTPUT_DIRECTORY = 'C:\\Users\\%s\\Desktop\\kernings' % USER_NAME

if not os.path.exists(PATH_TO_OUTPUT_DIRECTORY):
    os.makedirs(PATH_TO_OUTPUT_DIRECTORY)

font_file_names = os.listdir(PATH_TO_FONTS_FOLDER)
for font_file_name in font_file_names:
    if not font_file_name.endswith('ttf'):
        continue

    path_to_font = os.path.join(PATH_TO_FONTS_FOLDER, font_file_name)
    font = fontforge.open(path_to_font)

    font_dict = {
        'family_name': font.familyname,
        'em_size': font.em,
        'is_quadratic': font.is_quadratic,
    }

    glyph_names_to_unicode = {}
    for glyph_name in font:
        glyph_names_to_unicode[glyph_name] = font[glyph_name].unicode

    glyphs = {}
    for glyph_name in font:
        if font[glyph_name].unicode >= 0:
            new_glyph = {
                'unicode': font[glyph_name].unicode,
                'name': glyph_name,
                'width': font[glyph_name].width
            }

            kerning_values = {}
            list_of_repositionings_or_substitutions = font[glyph_name].getPosSub("*")
            # print(list_of_repositionings_or_substitutions)
            if list_of_repositionings_or_substitutions:
                for repositioning_or_substitution_tuple in list_of_repositionings_or_substitutions:
                    if repositioning_or_substitution_tuple[1] != 'Pair':
                        continue

                    name_of_the_other_glyph = repositioning_or_substitution_tuple[2]
                    unicode_of_the_other_glyph = glyph_names_to_unicode[name_of_the_other_glyph]

                    current_glyph_repositioning = (
                        repositioning_or_substitution_tuple[3],
                        repositioning_or_substitution_tuple[4],
                        repositioning_or_substitution_tuple[5],
                        repositioning_or_substitution_tuple[6]
                    )
                    other_glyph_repositioning = (
                        repositioning_or_substitution_tuple[7],
                        repositioning_or_substitution_tuple[8],
                        repositioning_or_substitution_tuple[9],
                        repositioning_or_substitution_tuple[10]
                    )

                    kerning_values[unicode_of_the_other_glyph] = (
                        current_glyph_repositioning,
                        other_glyph_repositioning
                    )
            new_glyph['kerns'] = kerning_values

            glyphs[font[glyph_name].unicode] = new_glyph
    font_dict['glyphs'] = glyphs
    font.close()

    path_to_output_file = os.path.join(PATH_TO_OUTPUT_DIRECTORY, "%s.json" % font_file_name[:-4])
    with open(path_to_output_file, "w") as output_file:
        output_file.write(json.dumps(font_dict))
@ctrlcctrlv
Copy link
Member

Your Metrics View image shows W but your code snippet image shows w - is that intentional? It's confusing.

@NathanWailes
Copy link
Author

@ctrlcctrlv Sorry about that; fixed. None of the glyphs had kerning information in the file that was created.

@ctrlcctrlv
Copy link
Member

Sorry, your Python code is also confusing. The screenshot shows CormorantUpright-Regular, while the font tests for CabinVFBeta-Italic.ttf

Your code is very…interesting, by the way…weve_reached_the_point_to_start is well…no sugarcoating, bad practice.

@NathanWailes
Copy link
Author

@ctrlcctrlv That was a quick hack because the program crashed on a particular font without first prompting me with an error, and I didn't want to wait for the program to repeat work for the fonts prior to the one that crashed. When I reran the code the program didn't crash, so I'm still confused about what happened.

@ctrlcctrlv
Copy link
Member

Assuming your code works, the actual FontForge call you're asking about is font[glyph_name].getPosSub("*")...can we see some clear examples of:

  • the font file
  • a kern that shows in the interface
  • a kern that shows up in font[glyph_name].getPosSub("*")

and

  • the font file
  • a kern that shows in the interface
  • a kern that doesn't show up in font[glyph_name].getPosSub("*")

@ctrlcctrlv
Copy link
Member

I understand that but you're asking us for help, meaning we have to read and run your code, so it would benefit all of us to have as few hacks as possible, it's just etiquette. I'm not being paid to help you, I volunteer to fix FontForge bugs, and I'd like to understand the bug as quickly as possible without hitting things like weve_reached_the_point_to_start.

@NathanWailes
Copy link
Author

NathanWailes commented Jul 21, 2019

I'd like to understand the bug as quickly as possible without hitting things like weve_reached_the_point_to_start

Understood.


To clarify:

Why is "the font file" bolded above and show up twice?
Why does "a kern that shows in the interface" show up twice above?

@ctrlcctrlv
Copy link
Member

Because that's what I need at minimum to help you. Two font files, one with a kern that shows in the interface and in Python, and one that shows in the interface and not in Python.

@NathanWailes
Copy link
Author

ok got it, working on that now.

@NathanWailes
Copy link
Author

NathanWailes commented Jul 21, 2019

Here's a link to the three fonts used below:
https://www.dropbox.com/sh/3nxy1wklyafyz41/AADHd5fcMF5wTtEAI2Gn-q8aa?dl=0

Screenshots of the FontForge UI showing the kerning for all three fonts:
Itim-Regular:
image

AlegreyaSC-Medium:
image

CormorantUpright-Regular:
image

Here's the example program used to generate the output shown further down:

import fontforge


# Working:
PATH_TO_FONT = 'C:\\Users\\Nathan\\Desktop\\examples\\Itim-Regular.ttf'
font = fontforge.open(PATH_TO_FONT)

repositioning_or_substitution = font['W'].getPosSub("*")
print(repositioning_or_substitution)

font.close()


print('\n---\n')


# Not working:
PATH_TO_FONT = 'C:\\Users\\Nathan\\Desktop\\examples\\AlegreyaSC-Medium.ttf'
font = fontforge.open(PATH_TO_FONT)

repositioning_or_substitution = font['W'].getPosSub("*")
print(repositioning_or_substitution)

font.close()


print('\n---\n')


# Not working:
PATH_TO_FONT = 'C:\\Users\\Nathan\\Desktop\\examples\\CormorantUpright-Regular.ttf'
font = fontforge.open(PATH_TO_FONT)

repositioning_or_substitution = font['W'].getPosSub("*")
print(repositioning_or_substitution)

font.close()

Here's the output from running the program:

>>> script = open( "C:\\Users\\Nathan\\Documents\\rhymecraft\\server\\services\\generate_video_for_lyrics\\fontforge_example.py", "r" )
>>> exec script
The following table(s) in the font have been ignored by FontForge
  Ignoring 'DSIG' digital signature table
The glyph named Delta is mapped to U+0394.
But its name indicates it should be mapped to U+2206.
The glyph named Omega is mapped to U+03A9.
But its name indicates it should be mapped to U+2126.
The glyph named mu is mapped to U+03BC.
But its name indicates it should be mapped to U+00B5.
(('Single Substitution lookup 52 subtable', 'Substitution', 'W.ss02'), ('Single Substitution lookup 51 subtable', 'Substitution', 'W.ss01'), ('Single Substitution lookup 50 subtable', 'Substitution', 'W.ss02'), ('Single Substitution lookup 49 subtable', 'Substitution', 'W.ss01'), ('Single Substitution lookup 48 subtable', 'Substitution', 'W.ss02'), ('Single Substitution lookup 47 subtable', 'Substitution', 'W.ss01'), ('Single Substitution lookup 46 subtable', 'Substitution', 'W.ss02'), ('Single Substitution lookup 45 subtable', 'Substitution', 'W.ss01'), ('Single Substitution lookup 44 subtable', 'Substitution', 'W.ss02'), ('Single Substitution lookup 43 subtable', 'Substitution', 'W.ss01'), ('Single Substitution lookup 42 subtable', 'Substitution', 'W.ss02'), ('Single Substitution lookup 41 subtable', 'Substitution', 'W.ss01'), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni210A', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'zdotaccent', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'zcaron', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'zacute', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'z', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ygrave', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1E8F', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ydieresis', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ycircumflex', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'yacute', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'y', 0, 0, -9, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'x', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'w', 0, 0, -6, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'v', 0, 0, -6, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1E63', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1E61', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'scommaaccent', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'scircumflex', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'scedilla', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'scaron', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'sacute', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 's', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'q', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'oe', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'otilde', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'oslashacute', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'oslash', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'omacron', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ohungarumlaut', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EE1', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EDF', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EDD', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EE3', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EDB', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ohorn', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1ECF', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ograve', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1ECD', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'odieresis', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1ED7', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1ED5', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1ED3', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1ED9', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1ED1', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ocircumflex', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni01D2', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'obreve', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'oacute', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'o', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1E21', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'gdotaccent', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'gcommaaccent', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'gcircumflex', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'gcaron', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'gbreve', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'g', 0, 0, -27, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EBD', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'eogonek', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'emacron', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EBB', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'egrave', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB9', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'edotaccent', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'edieresis', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EC5', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EC3', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EC1', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EC7', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EBF', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ecircumflex', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ecaron', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ebreve', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'eacute', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'e', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1E0F', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1E0D', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'dcroat', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'dcaron', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'eth', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'd', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'cdotaccent', 0, 0, -32, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ccircumflex', 0, 0, -32, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ccedilla', 0, 0, -32, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ccaron', 0, 0, -32, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'cacute', 0, 0, -32, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'c', 0, 0, -32, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'aeacute', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'ae', 0, 0, -34, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'atilde', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'aringacute', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'aring', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'aogonek', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'amacron', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA3', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'agrave', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA1', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'adieresis', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EAB', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA9', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA7', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EAD', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA5', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'acircumflex', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni01CE', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB5', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB3', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB1', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB7', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EAF', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'abreve', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'aacute', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'a', 0, 0, -36, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'O', 0, 0, -5, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Jcircumflex', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'J', 0, 0, -18, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Atilde', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Aringacute', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Aring', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Aogonek', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Amacron', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA2', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Agrave', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA0', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Adieresis', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EAA', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA8', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA6', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EAC', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EA4', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Acircumflex', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB4', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB2', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB0', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EB6', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'uni1EAE', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Abreve', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'Aacute', 0, 0, -11, 0, 0, 0, 0, 0), ("'kern' Horizontal Kerning lookup 0 per glyph data 0", 'Pair', 'A', 0, 0, -11, 0, 0, 0, 0, 0))

---

The following table(s) in the font have been ignored by FontForge
  Ignoring 'DSIG' digital signature table
Warning: Mac string is a subset of the Windows string in the 'name' table
 for the Family string in the English (US) language.
Warning: Mac and Windows entries in the 'name' table differ for the
 SubFamily string in the language English (US)
 Mac String: Medium
Windows String: Regular
(("'cpsp' Capital Spacing lookup 0 subtable", 'Position', 10, 0, 20, 0),)

---

The following table(s) in the font have been ignored by FontForge
  Ignoring 'DSIG' digital signature table
(("'cpsp' Capital Spacing in Latin lookup 0 subtable", 'Position', 5, 0, 10, 0),)

@ctrlcctrlv
Copy link
Member

OK, thanks for that. I can see what's going on.

Your problem is simple, your script only understands pair positioning (kern), but that's not the only type of positioning that can exist in a font. As you've seen, cpsp affects the output via single positioning.

So you have two choices:

  1. Update your script to understand single positioning, helpfully marked by FontForge with 'Position'.
  2. Create a new Python scripting API that will take two glyphs, and use the same algorithm as the Metrics View to give an output positioning Δx and Δy† as an integer.

I have a lot on my plate, but (2) seems interesting to me. I'll take a glance at what would be required to implement it, and if it's simple enough, perhaps open a PR real quick for you.

† It is perfectly valid for GPOS to affect vertical position, see curs OT table. Without this feature, creating fonts for e.g. Nastaliq script would be impossible.

@NathanWailes
Copy link
Author

NathanWailes commented Jul 21, 2019

Thank you for looking at this so quickly.

If you end up not having time for option 2, please consider giving me some links to more information that could help me to implement option 1. I just tried searching the docs for "single positioning" and only came up with two links, both of which were kind of confusing to understand.

@ctrlcctrlv
Copy link
Member

I took the glance I promised. It seems like the best way to implement would be to port ApplyTickedFeatures (features.c) to the Python API. We'd also most likely want to have a function that will give you the "default" features (what FontForge thinks most programs will apply without special intervention from the user, so all REQ features, kern, liga, ccmp, etc), then parse the opentype_str struct into a Python dict. Does this sound right to you @skef?

@NathanWailes Feel free to email me privately if you'd like me to do this for you for hire. That will of course bring it to №1 on my list. Until then, sorry, it's getting late in my country, and I might be going to another city tomorrow, so good night. I'm not sure when I'll get to this otherwise, but if @skef seems interested it might happen regardless, but of course not as fast as if you hire me 😛 I certainly don't have time to do your research for you.

@skef
Copy link
Contributor

skef commented Jul 21, 2019

There are a number of situations where some interface to ApplyTickedFeatures from scripting would be a good thing to have. Arguably this is one of them. But if I were @NathanWailes, given the unusual bulk-processing problem space, I would first try to hack around the problem.

(An aside: Single positioning can be a factor but I suspect the bulk of the problem here is not processing character-class based kerning.)

@NathanWailes: Suppose that instead of processing each file as it is you were to open it, generate() it as a ttf with old-kern and not opentype, open that just-generated file, and examine its kerning tables as you do now. That way FontForge would be "forced" to unroll the kerning data into a pairwise form. See this test script using AlegreyaSC-Medium.ttf as an example (the function is just a slightly modified version of your code above):

#!/usr/bin/env python

import fontforge
import os

tmpname = 'foo.ttf'

def process_font(font):
    glyphs = {}
    glyph_names_to_unicode = {}
    for glyph_name in font:
        glyph_names_to_unicode[glyph_name] = font[glyph_name].unicode
    for glyph_name in font:
        if font[glyph_name].unicode >= 0:
            new_glyph = {
                'unicode': font[glyph_name].unicode,
                'name': glyph_name,
                'width': font[glyph_name].width
            }

            kerning_values = {}
            list_of_repositionings_or_substitutions = font[glyph_name].getPosSub("*")
            # print(list_of_repositionings_or_substitutions)
            if list_of_repositionings_or_substitutions:
                for repositioning_or_substitution_tuple in list_of_repositionings_or_substitutions:
                    if repositioning_or_substitution_tuple[1] != 'Pair':
                        continue

                    name_of_the_other_glyph = repositioning_or_substitution_tuple[2]
                    unicode_of_the_other_glyph = glyph_names_to_unicode[name_of_the_other_glyph]

                    current_glyph_repositioning = (
                        repositioning_or_substitution_tuple[3],
                        repositioning_or_substitution_tuple[4],
                        repositioning_or_substitution_tuple[5],
                        repositioning_or_substitution_tuple[6]
                    )
                    other_glyph_repositioning = (
                        repositioning_or_substitution_tuple[7],
                        repositioning_or_substitution_tuple[8],
                        repositioning_or_substitution_tuple[9],
                        repositioning_or_substitution_tuple[10]
                    )
                    kerning_values[unicode_of_the_other_glyph] = (
                        current_glyph_repositioning,
                        other_glyph_repositioning
                    )
            new_glyph['kerns'] = kerning_values

            glyphs[font[glyph_name].unicode] = new_glyph
    return glyphs

#f = fontforge.open('Itim-Regular.ttf')
f = fontforge.open('AlegreyaSC-Medium.ttf')

fd = process_font(f)
print(fd[87])

print("\n\n")

f.generate(tmpname, flags='old-kern')
f.close()
fg = fontforge.open(tmpname)

fgd = process_font(fg)
print(fgd[87])

fg.close()

os.remove(tmpname)

Output:

$ ./test.pl
The following table(s) in the font have been ignored by FontForge
  Ignoring 'DSIG' digital signature table
Warning: Mac string is a subset of the Windows string in the 'name' table
 for the Family string in the English (US) language.
Warning: Mac and Windows entries in the 'name' table differ for the
 SubFamily string in the language English (US)
 Mac String: Medium
Windows String: Regular
{'unicode': 87, 'name': 'W', 'width': 946, 'kerns': {}}



Note: On Windows many apps can have problems with this font's kerning, because 27935 of its glyph kern pairs cannot be mapped to unicode-BMP kern pairs (eg, they have a Unicode value of -1) To avoid this, go to Generate, Options, and check the "Windows-compatible 'kern'" option.
{'unicode': 87, 'name': 'W', 'width': 946, 'kerns': {-1: ((0, 0, -14, 0), (0, 0, 0, 0)), 8230: ((0, 0, -96, 0), (0, 0, 0, 0)), 8229: ((0, 0, -96, 0), (0, 0, 0, 0)), 8228: ((0, 0, -96, 0), (0, 0, 0, 0)), 8226: ((0, 0, -76, 0), (0, 0, 0, 0)), 8222: ((0, 0, -96, 0), (0, 0, 0, 0)), 8218: ((0, 0, -96, 0), (0, 0, 0, 0)), 8212: ((0, 0, -76, 0), (0, 0, 0, 0)), 8211: ((0, 0, -76, 0), (0, 0, 0, 0)), 8171: ((0, 0, 24, 0), (0, 0, 0, 0)), 8170: ((0, 0, 24, 0), (0, 0, 0, 0)), 8169: ((0, 0, 24, 0), (0, 0, 0, 0)), 8168: ((0, 0, 24, 0), (0, 0, 0, 0)), 8167: ((0, 0, 24, 0), (0, 0, 0, 0)), 8166: ((0, 0, 24, 0), (0, 0, 0, 0)), 8163: ((0, 0, 24, 0), (0, 0, 0, 0)), 8162: ((0, 0, 24, 0), (0, 0, 0, 0)), 8161: ((0, 0, 24, 0), (0, 0, 0, 0)), 8160: ((0, 0, 24, 0), (0, 0, 0, 0)), 8124: ((0, 0, -46, 0), (0, 0, 0, 0)), 8121: ((0, 0, -46, 0), (0, 0, 0, 0)), 8120: ((0, 0, -46, 0), (0, 0, 0, 0)), 8119: ((0, 0, -46, 0), (0, 0, 0, 0)), 8118: ((0, 0, -46, 0), (0, 0, 0, 0)), 8116: ((0, 0, -46, 0), (0, 0, 0, 0)), 8115: ((0, 0, -46, 0), (0, 0, 0, 0)), 8114: ((0, 0, -46, 0), (0, 0, 0, 0)), 8113: ((0, 0, -46, 0), (0, 0, 0, 0)), 8112: ((0, 0, -46, 0), (0, 0, 0, 0)), 8071: ((0, 0, -46, 0), (0, 0, 0, 0)), 8070: ((0, 0, -46, 0), (0, 0, 0, 0)), 8069: ((0, 0, -46, 0), (0, 0, 0, 0)), 8068: ((0, 0, -46, 0), (0, 0, 0, 0)), 8067: ((0, 0, -46, 0), (0, 0, 0, 0)), 8066: ((0, 0, -46, 0), (0, 0, 0, 0)), 8065: ((0, 0, -46, 0), (0, 0, 0, 0)), 8064: ((0, 0, -46, 0), (0, 0, 0, 0)), 8059: ((0, 0, 24, 0), (0, 0, 0, 0)), 8058: ((0, 0, 24, 0), (0, 0, 0, 0)), 8057: ((0, 0, -30, 0), (0, 0, 0, 0)), 8056: ((0, 0, -30, 0), (0, 0, 0, 0)), 8049: ((0, 0, -46, 0), (0, 0, 0, 0)), 8048: ((0, 0, -46, 0), (0, 0, 0, 0)), 8031: ((0, 0, 24, 0), (0, 0, 0, 0)), 8029: ((0, 0, 24, 0), (0, 0, 0, 0)), 8027: ((0, 0, 24, 0), (0, 0, 0, 0)), 8025: ((0, 0, 24, 0), (0, 0, 0, 0)), 8023: ((0, 0, 24, 0), (0, 0, 0, 0)), 8022: ((0, 0, 24, 0), (0, 0, 0, 0)), 8021: ((0, 0, 24, 0), (0, 0, 0, 0)), 8020: ((0, 0, 24, 0), (0, 0, 0, 0)), 8019: ((0, 0, 24, 0), (0, 0, 0, 0)), 8018: ((0, 0, 24, 0), (0, 0, 0, 0)), 8017: ((0, 0, 24, 0), (0, 0, 0, 0)), 8016: ((0, 0, 24, 0), (0, 0, 0, 0)), 8005: ((0, 0, -30, 0), (0, 0, 0, 0)), 8004: ((0, 0, -30, 0), (0, 0, 0, 0)), 8003: ((0, 0, -30, 0), (0, 0, 0, 0)), 8002: ((0, 0, -30, 0), (0, 0, 0, 0)), 8001: ((0, 0, -30, 0), (0, 0, 0, 0)), 8000: ((0, 0, -30, 0), (0, 0, 0, 0)), 7943: ((0, 0, -46, 0), (0, 0, 0, 0)), 7942: ((0, 0, -46, 0), (0, 0, 0, 0)), 7941: ((0, 0, -46, 0), (0, 0, 0, 0)), 7940: ((0, 0, -46, 0), (0, 0, 0, 0)), 7939: ((0, 0, -46, 0), (0, 0, 0, 0)), 7938: ((0, 0, -46, 0), (0, 0, 0, 0)), 7937: ((0, 0, -46, 0), (0, 0, 0, 0)), 7936: ((0, 0, -46, 0), (0, 0, 0, 0)), 7929: ((0, 0, 24, 0), (0, 0, 0, 0)), 7928: ((0, 0, 24, 0), (0, 0, 0, 0)), 7927: ((0, 0, 24, 0), (0, 0, 0, 0)), 7926: ((0, 0, 24, 0), (0, 0, 0, 0)), 7925: ((0, 0, 24, 0), (0, 0, 0, 0)), 7924: ((0, 0, 24, 0), (0, 0, 0, 0)), 7923: ((0, 0, 24, 0), (0, 0, 0, 0)), 7922: ((0, 0, 24, 0), (0, 0, 0, 0)), 7907: ((0, 0, -30, 0), (0, 0, 0, 0)), 7906: ((0, 0, -30, 0), (0, 0, 0, 0)), 7905: ((0, 0, -30, 0), (0, 0, 0, 0)), 7904: ((0, 0, -30, 0), (0, 0, 0, 0)), 7903: ((0, 0, -30, 0), (0, 0, 0, 0)), 7902: ((0, 0, -30, 0), (0, 0, 0, 0)), 7901: ((0, 0, -30, 0), (0, 0, 0, 0)), 7900: ((0, 0, -30, 0), (0, 0, 0, 0)), 7899: ((0, 0, -30, 0), (0, 0, 0, 0)), 7898: ((0, 0, -30, 0), (0, 0, 0, 0)), 7897: ((0, 0, -30, 0), (0, 0, 0, 0)), 7896: ((0, 0, -30, 0), (0, 0, 0, 0)), 7895: ((0, 0, -30, 0), (0, 0, 0, 0)), 7894: ((0, 0, -30, 0), (0, 0, 0, 0)), 7893: ((0, 0, -30, 0), (0, 0, 0, 0)), 7892: ((0, 0, -30, 0), (0, 0, 0, 0)), 7891: ((0, 0, -30, 0), (0, 0, 0, 0)), 7890: ((0, 0, -30, 0), (0, 0, 0, 0)), 7889: ((0, 0, -30, 0), (0, 0, 0, 0)), 7888: ((0, 0, -30, 0), (0, 0, 0, 0)), 7887: ((0, 0, -30, 0), (0, 0, 0, 0)), 7886: ((0, 0, -30, 0), (0, 0, 0, 0)), 7885: ((0, 0, -30, 0), (0, 0, 0, 0)), 7884: ((0, 0, -30, 0), (0, 0, 0, 0)), 7863: ((0, 0, -46, 0), (0, 0, 0, 0)), 7862: ((0, 0, -46, 0), (0, 0, 0, 0)), 7861: ((0, 0, -46, 0), (0, 0, 0, 0)), 7860: ((0, 0, -46, 0), (0, 0, 0, 0)), 7859: ((0, 0, -46, 0), (0, 0, 0, 0)), 7858: ((0, 0, -46, 0), (0, 0, 0, 0)), 7857: ((0, 0, -46, 0), (0, 0, 0, 0)), 7856: ((0, 0, -46, 0), (0, 0, 0, 0)), 7855: ((0, 0, -46, 0), (0, 0, 0, 0)), 7854: ((0, 0, -46, 0), (0, 0, 0, 0)), 7853: ((0, 0, -46, 0), (0, 0, 0, 0)), 7852: ((0, 0, -46, 0), (0, 0, 0, 0)), 7851: ((0, 0, -46, 0), (0, 0, 0, 0)), 7850: ((0, 0, -46, 0), (0, 0, 0, 0)), 7849: ((0, 0, -46, 0), (0, 0, 0, 0)), 7848: ((0, 0, -46, 0), (0, 0, 0, 0)), 7847: ((0, 0, -46, 0), (0, 0, 0, 0)), 7846: ((0, 0, -46, 0), (0, 0, 0, 0)), 7845: ((0, 0, -46, 0), (0, 0, 0, 0)), 7844: ((0, 0, -46, 0), (0, 0, 0, 0)), 7843: ((0, 0, -46, 0), (0, 0, 0, 0)), 7842: ((0, 0, -46, 0), (0, 0, 0, 0)), 7841: ((0, 0, -46, 0), (0, 0, 0, 0)), 7840: ((0, 0, -46, 0), (0, 0, 0, 0)), 7823: ((0, 0, 24, 0), (0, 0, 0, 0)), 7822: ((0, 0, 24, 0), (0, 0, 0, 0)), 7785: ((0, 0, -14, 0), (0, 0, 0, 0)), 7784: ((0, 0, -14, 0), (0, 0, 0, 0)), 7783: ((0, 0, -14, 0), (0, 0, 0, 0)), 7782: ((0, 0, -14, 0), (0, 0, 0, 0)), 7781: ((0, 0, -14, 0), (0, 0, 0, 0)), 7780: ((0, 0, -14, 0), (0, 0, 0, 0)), 7779: ((0, 0, -14, 0), (0, 0, 0, 0)), 7778: ((0, 0, -14, 0), (0, 0, 0, 0)), 7777: ((0, 0, -14, 0), (0, 0, 0, 0)), 7776: ((0, 0, -14, 0), (0, 0, 0, 0)), 7763: ((0, 0, -30, 0), (0, 0, 0, 0)), 7762: ((0, 0, -30, 0), (0, 0, 0, 0)), 7761: ((0, 0, -30, 0), (0, 0, 0, 0)), 7760: ((0, 0, -30, 0), (0, 0, 0, 0)), 7759: ((0, 0, -30, 0), (0, 0, 0, 0)), 7758: ((0, 0, -30, 0), (0, 0, 0, 0)), 7757: ((0, 0, -30, 0), (0, 0, 0, 0)), 7756: ((0, 0, -30, 0), (0, 0, 0, 0)), 7747: ((0, 0, -10, 0), (0, 0, 0, 0)), 7746: ((0, 0, -10, 0), (0, 0, 0, 0)), 7713: ((0, 0, -30, 0), (0, 0, 0, 0)), 7712: ((0, 0, -30, 0), (0, 0, 0, 0)), 7689: ((0, 0, -30, 0), (0, 0, 0, 0)), 7688: ((0, 0, -30, 0), (0, 0, 0, 0)), 1307: ((0, 0, -30, 0), (0, 0, 0, 0)), 1306: ((0, 0, -30, 0), (0, 0, 0, 0)), 1267: ((0, 0, 24, 0), (0, 0, 0, 0)), 1265: ((0, 0, 24, 0), (0, 0, 0, 0)), 1263: ((0, 0, 24, 0), (0, 0, 0, 0)), 1257: ((0, 0, -30, 0), (0, 0, 0, 0)), 1256: ((0, 0, -30, 0), (0, 0, 0, 0)), 1255: ((0, 0, -30, 0), (0, 0, 0, 0)), 1254: ((0, 0, -30, 0), (0, 0, 0, 0)), 1247: ((0, 0, -8, 0), (0, 0, 0, 0)), 1246: ((0, 0, -8, 0), (0, 0, 0, 0)), 1241: ((0, 0, -30, 0), (0, 0, 0, 0)), 1240: ((0, 0, -30, 0), (0, 0, 0, 0)), 1237: ((0, 0, -46, 0), (0, 0, 0, 0)), 1236: ((0, 0, -46, 0), (0, 0, 0, 0)), 1235: ((0, 0, -46, 0), (0, 0, 0, 0)), 1234: ((0, 0, -46, 0), (0, 0, 0, 0)), 1233: ((0, 0, -46, 0), (0, 0, 0, 0)), 1232: ((0, 0, -46, 0), (0, 0, 0, 0)), 1201: ((0, 0, 24, 0), (0, 0, 0, 0)), 1200: ((0, 0, 24, 0), (0, 0, 0, 0)), 1199: ((0, 0, 24, 0), (0, 0, 0, 0)), 1198: ((0, 0, 24, 0), (0, 0, 0, 0)), 1177: ((0, 0, -8, 0), (0, 0, 0, 0)), 1176: ((0, 0, -8, 0), (0, 0, 0, 0)), 1139: ((0, 0, -30, 0), (0, 0, 0, 0)), 1138: ((0, 0, -30, 0), (0, 0, 0, 0)), 1118: ((0, 0, 24, 0), (0, 0, 0, 0)), 1113: ((0, 0, -42, 0), (0, 0, 0, 0)), 1109: ((0, 0, -14, 0), (0, 0, 0, 0)), 1108: ((0, 0, -30, 0), (0, 0, 0, 0)), 1103: ((0, 0, -32, 0), (0, 0, 0, 0)), 1101: ((0, 0, -8, 0), (0, 0, 0, 0)), 1092: ((0, 0, -28, 0), (0, 0, 0, 0)), 1091: ((0, 0, 24, 0), (0, 0, 0, 0)), 1089: ((0, 0, -30, 0), (0, 0, 0, 0)), 1086: ((0, 0, -30, 0), (0, 0, 0, 0)), 1084: ((0, 0, -10, 0), (0, 0, 0, 0)), 1083: ((0, 0, -42, 0), (0, 0, 0, 0)), 1079: ((0, 0, -8, 0), (0, 0, 0, 0)), 1076: ((0, 0, -42, 0), (0, 0, 0, 0)), 1072: ((0, 0, -46, 0), (0, 0, 0, 0)), 1071: ((0, 0, -32, 0), (0, 0, 0, 0)), 1069: ((0, 0, -8, 0), (0, 0, 0, 0)), 1060: ((0, 0, -28, 0), (0, 0, 0, 0)), 1057: ((0, 0, -30, 0), (0, 0, 0, 0)), 1054: ((0, 0, -30, 0), (0, 0, 0, 0)), 1052: ((0, 0, -10, 0), (0, 0, 0, 0)), 1051: ((0, 0, -42, 0), (0, 0, 0, 0)), 1047: ((0, 0, -8, 0), (0, 0, 0, 0)), 1044: ((0, 0, -42, 0), (0, 0, 0, 0)), 1040: ((0, 0, -46, 0), (0, 0, 0, 0)), 1033: ((0, 0, -42, 0), (0, 0, 0, 0)), 1029: ((0, 0, -14, 0), (0, 0, 0, 0)), 1028: ((0, 0, -30, 0), (0, 0, 0, 0)), 1008: ((0, 0, -44, 0), (0, 0, 0, 0)), 981: ((0, 0, -72, 0), (0, 0, 0, 0)), 973: ((0, 0, 24, 0), (0, 0, 0, 0)), 972: ((0, 0, -30, 0), (0, 0, 0, 0)), 971: ((0, 0, 24, 0), (0, 0, 0, 0)), 968: ((0, 0, 24, 0), (0, 0, 0, 0)), 966: ((0, 0, -28, 0), (0, 0, 0, 0)), 965: ((0, 0, 24, 0), (0, 0, 0, 0)), 959: ((0, 0, -30, 0), (0, 0, 0, 0)), 956: ((0, 0, -10, 0), (0, 0, 0, 0)), 955: ((0, 0, -46, 0), (0, 0, 0, 0)), 952: ((0, 0, -30, 0), (0, 0, 0, 0)), 948: ((0, 0, -46, 0), (0, 0, 0, 0)), 945: ((0, 0, -46, 0), (0, 0, 0, 0)), 944: ((0, 0, 24, 0), (0, 0, 0, 0)), 940: ((0, 0, -46, 0), (0, 0, 0, 0)), 939: ((0, 0, 24, 0), (0, 0, 0, 0)), 936: ((0, 0, 24, 0), (0, 0, 0, 0)), 934: ((0, 0, -28, 0), (0, 0, 0, 0)), 933: ((0, 0, 24, 0), (0, 0, 0, 0)), 927: ((0, 0, -30, 0), (0, 0, 0, 0)), 924: ((0, 0, -10, 0), (0, 0, 0, 0)), 923: ((0, 0, -46, 0), (0, 0, 0, 0)), 920: ((0, 0, -30, 0), (0, 0, 0, 0)), 916: ((0, 0, -46, 0), (0, 0, 0, 0)), 913: ((0, 0, -46, 0), (0, 0, 0, 0)), 908: ((0, 0, -30, 0), (0, 0, 0, 0)), 903: ((0, 0, -76, 0), (0, 0, 0, 0)), 808: ((0, 0, -96, 0), (0, 0, 0, 0)), 807: ((0, 0, -96, 0), (0, 0, 0, 0)), 806: ((0, 0, -96, 0), (0, 0, 0, 0)), 803: ((0, 0, -96, 0), (0, 0, 0, 0)), 601: ((0, 0, -30, 0), (0, 0, 0, 0)), 563: ((0, 0, 24, 0), (0, 0, 0, 0)), 562: ((0, 0, 24, 0), (0, 0, 0, 0)), 561: ((0, 0, -30, 0), (0, 0, 0, 0)), 560: ((0, 0, -30, 0), (0, 0, 0, 0)), 557: ((0, 0, -30, 0), (0, 0, 0, 0)), 556: ((0, 0, -30, 0), (0, 0, 0, 0)), 555: ((0, 0, -30, 0), (0, 0, 0, 0)), 554: ((0, 0, -30, 0), (0, 0, 0, 0)), 537: ((0, 0, -14, 0), (0, 0, 0, 0)), 536: ((0, 0, -14, 0), (0, 0, 0, 0)), 527: ((0, 0, -30, 0), (0, 0, 0, 0)), 526: ((0, 0, -30, 0), (0, 0, 0, 0)), 525: ((0, 0, -30, 0), (0, 0, 0, 0)), 524: ((0, 0, -30, 0), (0, 0, 0, 0)), 515: ((0, 0, -46, 0), (0, 0, 0, 0)), 514: ((0, 0, -46, 0), (0, 0, 0, 0)), 513: ((0, 0, -46, 0), (0, 0, 0, 0)), 512: ((0, 0, -46, 0), (0, 0, 0, 0)), 511: ((0, 0, -30, 0), (0, 0, 0, 0)), 510: ((0, 0, -30, 0), (0, 0, 0, 0)), 509: ((0, 0, -46, 0), (0, 0, 0, 0)), 508: ((0, 0, -46, 0), (0, 0, 0, 0)), 507: ((0, 0, -46, 0), (0, 0, 0, 0)), 506: ((0, 0, -46, 0), (0, 0, 0, 0)), 491: ((0, 0, -30, 0), (0, 0, 0, 0)), 490: ((0, 0, -30, 0), (0, 0, 0, 0)), 487: ((0, 0, -30, 0), (0, 0, 0, 0)), 486: ((0, 0, -30, 0), (0, 0, 0, 0)), 466: ((0, 0, -30, 0), (0, 0, 0, 0)), 465: ((0, 0, -30, 0), (0, 0, 0, 0)), 462: ((0, 0, -46, 0), (0, 0, 0, 0)), 461: ((0, 0, -46, 0), (0, 0, 0, 0)), 417: ((0, 0, -30, 0), (0, 0, 0, 0)), 416: ((0, 0, -30, 0), (0, 0, 0, 0)), 399: ((0, 0, -30, 0), (0, 0, 0, 0)), 376: ((0, 0, 24, 0), (0, 0, 0, 0)), 375: ((0, 0, 24, 0), (0, 0, 0, 0)), 374: ((0, 0, 24, 0), (0, 0, 0, 0)), 353: ((0, 0, -14, 0), (0, 0, 0, 0)), 352: ((0, 0, -14, 0), (0, 0, 0, 0)), 351: ((0, 0, -14, 0), (0, 0, 0, 0)), 350: ((0, 0, -14, 0), (0, 0, 0, 0)), 349: ((0, 0, -14, 0), (0, 0, 0, 0)), 348: ((0, 0, -14, 0), (0, 0, 0, 0)), 347: ((0, 0, -14, 0), (0, 0, 0, 0)), 346: ((0, 0, -14, 0), (0, 0, 0, 0)), 339: ((0, 0, -30, 0), (0, 0, 0, 0)), 338: ((0, 0, -30, 0), (0, 0, 0, 0)), 337: ((0, 0, -30, 0), (0, 0, 0, 0)), 336: ((0, 0, -30, 0), (0, 0, 0, 0)), 335: ((0, 0, -30, 0), (0, 0, 0, 0)), 334: ((0, 0, -30, 0), (0, 0, 0, 0)), 333: ((0, 0, -30, 0), (0, 0, 0, 0)), 332: ((0, 0, -30, 0), (0, 0, 0, 0)), 291: ((0, 0, -30, 0), (0, 0, 0, 0)), 290: ((0, 0, -30, 0), (0, 0, 0, 0)), 289: ((0, 0, -30, 0), (0, 0, 0, 0)), 288: ((0, 0, -30, 0), (0, 0, 0, 0)), 287: ((0, 0, -30, 0), (0, 0, 0, 0)), 286: ((0, 0, -30, 0), (0, 0, 0, 0)), 285: ((0, 0, -30, 0), (0, 0, 0, 0)), 284: ((0, 0, -30, 0), (0, 0, 0, 0)), 269: ((0, 0, -30, 0), (0, 0, 0, 0)), 268: ((0, 0, -30, 0), (0, 0, 0, 0)), 267: ((0, 0, -30, 0), (0, 0, 0, 0)), 266: ((0, 0, -30, 0), (0, 0, 0, 0)), 265: ((0, 0, -30, 0), (0, 0, 0, 0)), 264: ((0, 0, -30, 0), (0, 0, 0, 0)), 263: ((0, 0, -30, 0), (0, 0, 0, 0)), 262: ((0, 0, -30, 0), (0, 0, 0, 0)), 261: ((0, 0, -46, 0), (0, 0, 0, 0)), 260: ((0, 0, -46, 0), (0, 0, 0, 0)), 259: ((0, 0, -46, 0), (0, 0, 0, 0)), 258: ((0, 0, -46, 0), (0, 0, 0, 0)), 257: ((0, 0, -46, 0), (0, 0, 0, 0)), 256: ((0, 0, -46, 0), (0, 0, 0, 0)), 255: ((0, 0, 24, 0), (0, 0, 0, 0)), 253: ((0, 0, 24, 0), (0, 0, 0, 0)), 248: ((0, 0, -30, 0), (0, 0, 0, 0)), 246: ((0, 0, -30, 0), (0, 0, 0, 0)), 245: ((0, 0, -30, 0), (0, 0, 0, 0)), 244: ((0, 0, -30, 0), (0, 0, 0, 0)), 243: ((0, 0, -30, 0), (0, 0, 0, 0)), 242: ((0, 0, -30, 0), (0, 0, 0, 0)), 231: ((0, 0, -30, 0), (0, 0, 0, 0)), 230: ((0, 0, -46, 0), (0, 0, 0, 0)), 229: ((0, 0, -46, 0), (0, 0, 0, 0)), 228: ((0, 0, -46, 0), (0, 0, 0, 0)), 227: ((0, 0, -46, 0), (0, 0, 0, 0)), 226: ((0, 0, -46, 0), (0, 0, 0, 0)), 225: ((0, 0, -46, 0), (0, 0, 0, 0)), 224: ((0, 0, -46, 0), (0, 0, 0, 0)), 221: ((0, 0, 24, 0), (0, 0, 0, 0)), 216: ((0, 0, -30, 0), (0, 0, 0, 0)), 214: ((0, 0, -30, 0), (0, 0, 0, 0)), 213: ((0, 0, -30, 0), (0, 0, 0, 0)), 212: ((0, 0, -30, 0), (0, 0, 0, 0)), 211: ((0, 0, -30, 0), (0, 0, 0, 0)), 210: ((0, 0, -30, 0), (0, 0, 0, 0)), 199: ((0, 0, -30, 0), (0, 0, 0, 0)), 198: ((0, 0, -46, 0), (0, 0, 0, 0)), 197: ((0, 0, -46, 0), (0, 0, 0, 0)), 196: ((0, 0, -46, 0), (0, 0, 0, 0)), 195: ((0, 0, -46, 0), (0, 0, 0, 0)), 194: ((0, 0, -46, 0), (0, 0, 0, 0)), 193: ((0, 0, -46, 0), (0, 0, 0, 0)), 192: ((0, 0, -46, 0), (0, 0, 0, 0)), 183: ((0, 0, -76, 0), (0, 0, 0, 0)), 121: ((0, 0, 24, 0), (0, 0, 0, 0)), 115: ((0, 0, -14, 0), (0, 0, 0, 0)), 113: ((0, 0, -30, 0), (0, 0, 0, 0)), 111: ((0, 0, -30, 0), (0, 0, 0, 0)), 109: ((0, 0, -10, 0), (0, 0, 0, 0)), 103: ((0, 0, -30, 0), (0, 0, 0, 0)), 99: ((0, 0, -30, 0), (0, 0, 0, 0)), 97: ((0, 0, -46, 0), (0, 0, 0, 0)), 95: ((0, 0, -96, 0), (0, 0, 0, 0)), 89: ((0, 0, 24, 0), (0, 0, 0, 0)), 83: ((0, 0, -14, 0), (0, 0, 0, 0)), 81: ((0, 0, -30, 0), (0, 0, 0, 0)), 79: ((0, 0, -30, 0), (0, 0, 0, 0)), 77: ((0, 0, -10, 0), (0, 0, 0, 0)), 71: ((0, 0, -30, 0), (0, 0, 0, 0)), 67: ((0, 0, -30, 0), (0, 0, 0, 0)), 65: ((0, 0, -46, 0), (0, 0, 0, 0)), 46: ((0, 0, -96, 0), (0, 0, 0, 0)), 45: ((0, 0, -76, 0), (0, 0, 0, 0)), 44: ((0, 0, -96, 0), (0, 0, 0, 0))}}

@skef skef removed the bite-sized label Jul 21, 2019
@skef
Copy link
Contributor

skef commented Jul 21, 2019

(Nothing with a new python interface and doc requirements is really "bite-sized". OpenType lookup processing has a lot of quirks.)

@NathanWailes
Copy link
Author

NathanWailes commented Jul 23, 2019

@skef Thank you for your help. I tried implementing your solution and while it seemed to be working at first, now the saves (to the temp font) keep failing on other fonts, and the code keeps crashing at some point starting at the "Cs" (in Google's list of free fonts). The window immediately closes, so I can't see what's happening. I tried putting in a broad try/except, and that doesn't even seem to work. Any idea what might be going on, or how I could try to further debug it?


To get the fonts, download Google's GitHub repo for the fonts, then do a search for every file that ends in "ttf" and move them into a new directory.


Here's the code:

# run the command below in the fontforge "embedded" python interpreter: C:\Program Files (x86)\FontForgeBuilds\bin\ffpython.exe
# exec open( "C:\\path\\to\\this\\file.py", "r" )
import json
import os

import fontforge

USER_NAME = 'Nathan'
PATH_TO_FONTS_FOLDER = 'C:\\Users\\Nathan\\Desktop\\fonts'
PATH_TO_OUTPUT_DIRECTORY = 'C:\\Users\\%s\\Desktop\\kernings' % USER_NAME

if not os.path.exists(PATH_TO_OUTPUT_DIRECTORY):
    os.makedirs(PATH_TO_OUTPUT_DIRECTORY)


def get_glyph_dicts(_font):
    glyphs = {}
    glyph_names_to_unicode = {}
    for glyph_name in _font:
        glyph_names_to_unicode[glyph_name] = _font[glyph_name].unicode
    for glyph_name in _font:
        if _font[glyph_name].unicode >= 0:
            new_glyph = {
                'unicode': _font[glyph_name].unicode,
                'name': glyph_name,
                'width': _font[glyph_name].width
            }

            kerning_values = {}
            list_of_repositionings_or_substitutions = _font[glyph_name].getPosSub("*")
            if list_of_repositionings_or_substitutions:
                for repositioning_or_substitution_tuple in list_of_repositionings_or_substitutions:
                    if repositioning_or_substitution_tuple[1] != 'Pair':
                        continue

                    name_of_the_other_glyph = repositioning_or_substitution_tuple[2]
                    unicode_of_the_other_glyph = glyph_names_to_unicode[name_of_the_other_glyph]

                    current_glyph_repositioning = (
                        repositioning_or_substitution_tuple[3],
                        repositioning_or_substitution_tuple[4],
                        repositioning_or_substitution_tuple[5],
                        repositioning_or_substitution_tuple[6]
                    )
                    other_glyph_repositioning = (
                        repositioning_or_substitution_tuple[7],
                        repositioning_or_substitution_tuple[8],
                        repositioning_or_substitution_tuple[9],
                        repositioning_or_substitution_tuple[10]
                    )
                    kerning_values[unicode_of_the_other_glyph] = (
                        current_glyph_repositioning,
                        other_glyph_repositioning
                    )
            new_glyph['kerns'] = kerning_values

            glyphs[_font[glyph_name].unicode] = new_glyph
    return glyphs


font_file_names = os.listdir(PATH_TO_FONTS_FOLDER)

weve_reached_the_font_name_to_start_at = False

for font_file_name in font_file_names:
    try:
        if not font_file_name.endswith('ttf'):
            continue
        if not weve_reached_the_font_name_to_start_at:
            if font_file_name != 'Cagliostro-Regular.ttf':
                continue
            else:
                weve_reached_the_font_name_to_start_at = True

        print(font_file_name)

        path_to_font = os.path.join(PATH_TO_FONTS_FOLDER, font_file_name)
        font = fontforge.open(path_to_font)

        font_dict = {
            'family_name': font.familyname,
            'em_size': font.em,
            'is_quadratic': font.is_quadratic
        }

        temporary_font_file_name = '%s_tmp.ttf' % font_file_name[:-4]
        try:
            font.generate(temporary_font_file_name, flags='old-kern')
        except EnvironmentError:
            continue
        font.close()

        tmp_font = fontforge.open(temporary_font_file_name)
        font_dict['glyphs'] = get_glyph_dicts(tmp_font)
        tmp_font.close()

        path_to_output_file = os.path.join(PATH_TO_OUTPUT_DIRECTORY, "%s.json" % font_file_name[:-4])
        with open(path_to_output_file, "w") as output_file:
            output_file.write(json.dumps(font_dict))

        try:
            os.remove(temporary_font_file_name)
        except:
            pass
    except:
        print('Hit an exception')

@skef
Copy link
Contributor

skef commented Jul 23, 2019

Oh, it's probably a memory leak in FontForge if I had to guess.

Assuming you have spare disk space try doing load/save first, in advance of the rest of it. Since you're only processing ttf files you can do something like (the following is pseudo-code)

for each filename:
    if (intermediate_directory/filename)
        continue
    f = fontforge.open(input_directory/filename)
    f.save(intermediate_directory/filename, flags=('old-kern')
    f.close()

That way if it crashes you can just run it a few times until it gets to the end, and then run something like your older script on that directory.

It could be that you'll also run into memory problems with that. Just add code to periodically save your results (maybe by pickling), and to track which files you've already processed so you can skip over them. The skipping process is unlikely to leak and you'll eventually get to the end.

@skef
Copy link
Contributor

skef commented Jul 23, 2019

As far as failed saves go, if that's not a memory problem we would need an example. But given that you're bulk processing, perhaps you don't need every single file?

@ctrlcctrlv
Copy link
Member

Just so you know @skef he asked me to help him get this working via Pango (apply PangoAttrList with colors to arbitrary text) for hire, I don't know why he's still replying here. I told him:

The thing is, I'm not even sure if having this kerning information will be
sufficient to solve my problem. To understand what I'm really trying to
accomplish, check out this:
python-pillow/Pillow#3977 Since you seem to have
some expertise with fonts, I'd be interested in your opinion of the best
way to solve this problem.

I however agree with you, you're just adding hack after hack. Adding
understanding of horizontal kerning to your app might solve the problem for
Latin text...but I still think you're going about this the wrong way, and
there's no way what you're doing, rendering in chunks like this, will work for
e.g. Arabic. Certain Latin handwriting fonts might also break.

You need to render everything at once all the time, then apply attributes like
color. It is probably not possible to do this in PIL alone...you should be
using Pango, and if that's not enough for you, HarfBuzz directly, to render
all the text, then select certain letters and change their color.

And I also started working on this, so I really don't think continuing this conversation is fruitful. I stand by my observation that FontForge can't do what he wants anyway unless you use FontForge to implement your own text rendering engine like Harfbuzz, which just seems silly to me.

@NathanWailes
Copy link
Author

@ctrlcctrlv Sorry, I didn't mean to waste skef's time. I just wanted to try his solution because I didn't know if you'd manage to get the Pango solution working, and then when I ran into an issue I figured my question would be quick for him to answer.

@ctrlcctrlv
Copy link
Member

OK I was literally working with pangocairocffi for the job you gave me when I saw these replies. In general, if you've already hired someone to help you, probably not the best to keep asking volunteers for help, especially since @skef's idea was always a huge hack not expected to work in every case

@NathanWailes
Copy link
Author

@ctrlcctrlv Understood; I thought it was just a quick question. I just wanted to try his solution quickly so that if the Pango solution didn't work, I could weigh this hacky approach against your more-in-depth (more expensive?) Harfbuzz approach. I strongly prefer a non-hacky solution but I'm just a freelance programmer and it isn't easy for me to save the money to hire other freelancers for large projects.

@ctrlcctrlv
Copy link
Member

(This is way off topic, but fixing FontForge to support all possible fonts is way more expensive than just using Pango, I already got it mostly working)

@skef
Copy link
Contributor

skef commented Jul 23, 2019

Just to be clear I offered that first option freely and always choose how I use my time here. As far as other arrangements go those are not my business and, accordingly, not my problem (other than a general hope that people deal well with one another).

Bulk data extraction is almost always hacky -- a script runs supervised and only needs to work once. If this were needed in an ongoing processing cycle it would be a different story.

@khaledhosny
Copy link
Contributor

I’m not sure what this issue is about, but Pillow already has support for complex text layout (including Latin) when Raqm integration is enabled. Refer to Pillow documentation on how to enable Raqm.

@NathanWailes
Copy link
Author

@khaledhosny Thank you for your advice! You can find an explanation of the underlying problem here: python-pillow/Pillow#3977 ctrlcctrlv showed me (in an email) how to use Pango (code below), but if it would be easier to use Raqm, I might prefer that.

Here's what I was going to go with (code from ctrlcctrlv):

import numpy
import cairocffi
import pangocffi
import pangocairocffi
def test_png():
    filename = 'test.png'
    width, height = 1000, 1000

    surface = cairocffi.ImageSurface(cairocffi.FORMAT_RGB24, width, height)

    context = cairocffi.Context(surface)
    layout = pangocairocffi.create_layout(context)
    layout.set_width(pangocffi.units_from_double(width))
    layout.set_alignment(pangocffi.Alignment.CENTER)
    layout.set_markup('<span color="#FFFFFF" font="80">ويكيبيديا\n</span><span font="80" color="#FFFFFF">ويكيب<span color="#00FFFF">يديا</span></span>')
    pangocairocffi.show_layout(context, layout)
    surface.write_to_png(filename)
    surface.finish()

test_png()

That code would produce the image below:
image

The code is really simple, which is great, but I've been running into some issues getting Pango and Cairo installed. However, ctrlcctrlv is helping me with that.

In any case, I don't think I need any further help; I just wanted to thank you for your input and give more background information in case you maybe had an opinion and didn't mind sharing it.

@ctrlcctrlv
Copy link
Member

ctrlcctrlv commented Jul 24, 2019

The problem, as I see it, is that:

  • Pillow simply does not support writing multi-colored text in one call. Am I wrong? Is there a way to write ويكيب in white and يديا in blue, with correct rendering as one word ويكيبيديا as happens in karaoke?
  • Hacks to get it to do so via multiple calls to Pango doomed to fail on edge cases, where "edge" includes entire languages. At most, Latin can be hacked around as long as you understand OpenType positioning (and legacy kern positioning) and are willing to work around it.
  • To get it to support writing, just as a singular example, Arabic, in multiple calls to the text renderer would require only using Pillow for rasterization and implementing your own shaper in pure Python, perhaps using FontForge for help parsing fonts. Certainly this is doable. But text rendering is hard to get right, and Pango is the standard.

@khaledhosny
Copy link
Contributor

Right, I don’t think Pillow supports this. Very few applications/libraries do actually, Pango stands out here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants