Making Linux send two Unicode points from one keypress

[iandoug]

Trying on an older PC, software (OS and KDE) is not up to date. But good enough for experimenting without the risk of locking me out of my main PC.

However the 'obvious' approach fails, and the confirmation is here:

https://unix.stackexchange.com/questions/43073/custom-xkb-layout-in-which-one-key-creates-two-unicode-code-points

I don't like the given answer, I will take a look if we can add custom keysyms somewhere, to avoid having to dedicate a key to combining-macron-below. Ian still thinks each variant of the vowels should be a single key press.

Just created this as a separate issue to track it better.

[HughP]

did you check out ibus for linux? It is not xkb but it is linux. https://wiki.archlinux.org/index.php/IBus

…

On Sat, Jun 30, 2018 at 8:11 AM, Ian Douglas ***@***.***> wrote: Trying on an older PC, software (OS and KDE) is not up to date. But good enough for experimenting without the risk of locking me out of my main PC. However the 'obvious' approach fails, and the confirmation is here: https://unix.stackexchange.com/questions/43073/custom- xkb-layout-in-which-one-key-creates-two-unicode-code-points I don't like the given answer, I will take a look if we can add custom keysyms somewhere, to avoid having to dedicate a key to combining-macron-below. Ian still thinks each variant of the vowels should be a single key press. Just created this as a separate issue to track it better. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACdJgOyE_Zi59Xc60ZjqcZE6sdem4k-ks5uB5UqgaJpZM4U-AMt> .

[iandoug]

Thanks, didn't know about that. Currently trying to work through various (ancient) docs about xkb, part of the (well-known) problem with xkb is the "difficult" and very old docs. Like KMail, it's one of those vital but unglamourous things that no one wants to work on.... :-)
I see the packages are available for Gentoo so will take a look.

[iandoug]

Just posting this here for future reference.

https://sites.google.com/site/brightmeasurement/Home/keyboard-for-english-around-the-word

Relevance: (well, in 2013)

1. Mexican keyboards in ISO form factor not ANSI. Tried to find pics of current laptops on sale in MX, they appear to be ANSI rather than ISO layouts.
2. Nigerian keyboard has a plethora of combining diacritics. Guess that's how they made one keyboard for all their languages. Surprised they didn't fix QWERTY while they were at it :-)

[HughP]

Yea, Mexico is a hodgepodge... officially it is an ISO keyboard region but there are so many cross-boarder imports from the USA that ANSI is really common... and then there is china too...

…

On Sun, Jul 1, 2018 at 8:09 AM, Ian Douglas ***@***.***> wrote: Just posting this here for future reference. https://sites.google.com/site/brightmeasurement/Home/ keyboard-for-english-around-the-word Relevance: (well, in 2013) 1. Mexican keyboards in ISO form factor not ANSI. Tried to find pics of current laptops on sale in MX, they appear to be ANSI rather than ISO layouts. 2. Nigerian keyboard has a plethora of combining diacritics. Guess that's how they made one keyboard for all their languages. Surprised they didn't fix QWERTY while they were at it :-) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACdJtZ8Iv1B5clVdv5CUaRvm3zE5HEeks5uCOZFgaJpZM4U-AMt> .

[iandoug]

Stumbled across this while poking around exploring Compose key functionality in Linux.

https://ubuntuforums.org/showthread.php?t=2384387
Although it does seem like using a hammer to thread a needle. Must be more elegant solutions. Still researching xkb, but work etc got in the way.

Also stumbled across things like ų ą ǫ etc and was wondering why they picked a macron below when the other American tribes used the oganek. Suppose one is for tone and the other nasal or somesuch?

[HughP]

Well, the thing is with orthographies, is that they generally follow the pattern of the orthography that the missionary was most familiar with... hence Russian has Greek letters. I just replaced the ANSI keyboard on my laptop with an ISO keyboard and am now looking to modify my own XKB keyboards. In the process I ran across these: to edit the native Linux XKB keyboards, here is a good guide: http://linux.lsdev.sil.org/wiki/index.php/Building_an_XKB_Keyboard The link at the top of the page is also a classic and recommended read: http://www.charvolant.org/~doug/xkb/

…

On Thu, Jul 5, 2018 at 4:25 PM, Ian Douglas ***@***.***> wrote: Stumbled across this while poking around exploring Compose key functionality in Linux. https://ubuntuforums.org/showthread.php?t=2384387 Although it does seem like using a hammer to thread a needle. Must be more elegant solutions. Still researching xkb, but work etc got in the way. Also stumbled across things like ų ą ǫ etc and was wondering why they picked a macron below when the other American tribes used the oganek. Suppose one is for tone and the other nasal or somesuch? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACdJlBSglKgKObm5QEHFLG_Kj8nIpkRks5uDqBggaJpZM4U-AMt> .

[iandoug]

Thanks, didn't know about the SIL page.
I have played with xkb in the past, and successfully loaded custom layouts, but what we need to do in this case is a bit different, i.e. duping it into sending two Unicode points where is it "designed" to send one only.
Part of the problem is that xkb is very old and there are different ways of loading the layout, especially what works in X will probably not work in terminal-when-you-boot, so you need something for that too... especially if you are logging in and your password uses diacritics. (FWIW, even my laptop boots to the command prompt rather than KDE ... just another obstacle in the way of whoever stole my previous laptop)

Some other links you may or may not know about: (currently open tabs... :-) )
https://unix.stackexchange.com/questions/43073/custom-xkb-layout-in-which-one-key-creates-two-unicode-code-points
Think we need to make a change to this file, and then force X to load the changes:
https://cgit.freedesktop.org/xorg/proto/x11proto/tree/XF86keysym.h

https://askubuntu.com/questions/93772/where-do-i-find-a-list-of-all-x-keysyms-these-days
https://wiki.archlinux.org/index.php/xmodmap
https://www.tldp.org/HOWTO/Intkeyb/x336.html
https://askubuntu.com/questions/325272/permanent-xmodmap-in-ubuntu-13-04
https://wiki.archlinux.org/index.php/X_keyboard_extension
https://www.x.org/wiki/XKB/ (old links)

https://help.ubuntu.com/community/Custom%20keyboard%20layout%20definitions?action=show&redirect=Howto%3A+Custom+keyboard+layout+definitions

Last night I was playing with the "Compose" function, and it looks like they have mapped both hyphen-minus and underscore to produce a top macron (U+00AF). So I was wondering if we could modify that so that the underscore produces a U+02CD instead ... and then hope xkb morphs that into a combining low macron.

On the plus side: fewer dedicated keys needed on layout.
On the negative side: typing will still be a pain, and KLA can't handle Compose key sequences.

[iandoug]

Just discovered that GTK has their OWN mapping tables for Compose key (which are HARD CODED)... just to muddy the waters even further.
My compose sequences won't work in Kate (my text editor) which is why I thought the function was broken. But they work okay (most of them) in LibreOffice. And here in Firefox. But not in Konsole or Kmail .... so not sure if I'm using GTK's version or not ... might explain why some things work as expected and others don't.
But clearly Compose sequences as a solution to the diacritic problem for tcf or dnj is going to have issues.

[HughP]

That must be why SIL went with iBus... not sure the full reasons on that but there were hundreds of layouts in .kmn format already too.

…

On Fri, Jul 6, 2018 at 1:57 AM, Ian Douglas ***@***.***> wrote: Just discovered that GTK has their OWN mapping tables for Compose key... just to muddy the waters even further. My compose sequences won't work in Kate (my text editor) which is why I thought the function was broken. But they work okay (most of them) in LibreOffice. And here in Firefox. But not in Konsole or Kmail .... so not sure if I'm using GTK's version or not ... might explain why some things work as expected and others don't. But clearly Compose sequences as a solution to the diacritic problem for tcf or dnj is going to have issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACdJsSL_IQh4FaDweIt1FN94TboMSwZks5uDyZjgaJpZM4U-AMt> .

[iandoug]

[rant: tired of trying to make markdown do what I want. so formatting below not correct.[/rant]

FWIW just managed to get the compose sequences to work in Kate and KMail, by accident.
I created a custom ~/.XCompose file and put some custom things in:

<Multi_key>

: "π" U03C0 # Greek small letter pi
<Multi_key> : "τ" U03C4 # Greek small letter tau
<Multi_key> : "ℓ" U2113 # litre

and then noticed that those worked in Kate etc ... then remembered seeing I need to add

(hash) Import default rules from the system Compose file
include "%L"

at the top of the file, and Voila, suddenly it works.
Not sure why xkb is not loading them by default, may be something to do with EurKeys instead of US layout.
Will experiment further with xkb later.

[iandoug]

Have you had any joy with this yet?
I tried the first answer from
https://unix.stackexchange.com/questions/43073/custom-xkb-layout-in-which-one-key-creates-two-unicode-code-points

by modifying the latam config file as follows:

// key { [ bar, degree, notsign, notsign ] };
key { [XF86Launch1, XF86Launch2, q, Q ] };

and then ran xmodmap with this in an input file:
keysym XF86Launch1 = U0075 U0331
keysym XF86Launch2 = U0055 U0331

Which is, as best I can tell, what the instructions are saying to do (as a once-off not permanent fix).
Then switched layout from eu to latam and checked it was right layout by accessing various characters mapped to AltGr.

But pressing Tilde key does not work...
Mmm I see pressing it in Firefox give me upper U
Mmm 2: press once, get U, press shift-tilde, get macron below, so can get U̱ by two key presses.
That means it is mapping the combining macron to the shift function instead of sending two Unicode points on one keypress as advertised.

/usr/share/X11/xkb/symbols $ xmodmap -pk | grep 331
49 0x0055 (U) 0x1000331 (U0331)
156 0x0075 (u) 0x1000331 (U0331) 0x0075 (u) 0x1000331 (U0331) 0x0075 (u) 0x1000331 (U0331)
157 0x0055 (U) 0x1000331 (U0331) 0x0055 (U) 0x1000331 (U0331) 0x0055 (U) 0x1000331 (U0331)

Am vaguely kinda understanding what the printout above is saying, and it's not what I want.... looks like it put U as "normal" and combining belowmacron as "shift" on the tilde key, dropped the q Q, and added other definitions for u and U with combining macrons.

So I wonder if xkb behaves differently now to when the solution was written in 2012, or if I am doing something stupid ...

[iandoug]

Something just occurred to me ... we're trying to make X send effectively two characters on one key press.. but what about the application programs? Will they be able to handle such things? Maybe they all expect one keypress == one char / Unicode code point, and will barf (or only take the first) when sent a multi-char string?

I tried a few other things on the process above and still no joy. Maybe I'll have a bright idea in my dreams.

[iandoug]

You said something the other day about using AutoHotKey on Windows to do what we want... eventually stumbled across current Linux clone:
https://github.com/autokey/autokey
Will take a look tomorrow.

[iandoug]

Saving for future reference, no longer maintained but certainly claims to be able to send two+ code points with one key press
https://github.com/byrongibson/Pystromo/tree/master/pystromo

[iandoug]

Had email conversation with Sergey Udaltsov, maintainer of xkb config package.

His response:
"To the best of my knowledge, there is no straightforward and correct
way to send multiple unicode codepoints with one keypress using pure
xkb. I guess you might need to talk to some folks in the ibus
department.. Or any other input method, above xkb level in X."

Also asked about and tried using "actions" function in xkb, but that does not send key strokes, it sends "messages" so does not work. (well, I could not get it to work.)

And it seems that modifying X or xkb to handle more than one Unicode point per keypress will break things elsewhere, so "not gonna happen", best I can tell.

So currently going to look at ibus until something better comes along.

[iandoug]

hi Hugh

On Friday, 06 July 2018 3:48:36 AM SAST HughP wrote: Well, the thing is with orthographies, is that they generally follow the pattern of the orthography that the missionary was most familiar with... hence Russian has Greek letters.

Is there a 2-letter code for Mephaa? Thanks, Ian

…

-- [email protected] http://www.zti.co.za Zero 2 Infinity - The net.works Phone +27-21-975-7273

[HughP]

no, just the three letter code. in all it applications the three letter code is valid if there is no two letter code. See the IANA tag here: https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry Type: language Subtag: tcf Description: Malinaltepec Me'phaa Description: Malinaltepec Tlapanec Added: 2009-07-29 On Tue, Jul 10, 2018 at 11:34 AM, Ian Douglas <[email protected]> wrote:

…

hi Hugh On Friday, 06 July 2018 3:48:36 AM SAST HughP wrote: > Well, the thing is with orthographies, is that they generally follow the > pattern of the orthography that the missionary was most familiar with... > hence Russian has Greek letters. Is there a 2-letter code for Mephaa? Thanks, Ian -- ***@***.*** http://www.zti.co.za Zero 2 Infinity - The net.works Phone +27-21-975-7273 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACdJkDtLU9-5MiO34dHznIZ3i0FTsFmks5uFPOrgaJpZM4U-AMt> .

[iandoug]

Okay. Just asking because seeing instructions in xkb etc to use 2-letter codes.
Have installed ibus and made mapping (attached version 1), need to experiment a bit to see if it actually works. So far ibus seems rather buggy. I've been basically following the instructions here:
http://www.studymongolian.net/technical/how-to-create-linux-input-method-editor/

Game plan at the moment is to define an xkb layout using vowels with macron above, and use ibus to switch those to macron below. I can't see a better way at the moment. It also builds on what I did with KLA, where I used the same substitution.

Still busy with KLA -> KLE script ... need to put in code to auto-fill in missing upper/lower case as needed, move the labels on non-character keys to "normal" slot as opposed to "shift" slot where they typically are, and worry about making sure there is a left and right shift defined, and handle cases where they are on the same key.
It's a command-line PHP proggie at the moment but will probably make a web front end so that the user does not have to type a long string of options every time. At the moment it generates valid JSON which KLA accepts, after manually fixing the right shift key.

tcf1.txt

[iandoug]

Gave up on ibus ... did not like all the errors the script threw up. Also could not get it to work and not in the mood for a struggle. Anyway the solution should be "easy" even for non-technical people.

So tried autokey, which took a bit of playing around because there is no Gentoo package, and their current QT support is iffy. However managed to get the GTK version to run, and now it happily (if somewhat slowly) is able to replace a typed tilde character with "p". Just as a does-it-work test. Tested working in two KDE QT apps and two GTK apps.
Maybe must move their folder to the front of my path instead of the back.
So will go with this solution for now. Next steps is to create xkb file for Mephaa layout. I see you are using v9 of the layouts I did, I will fix v8 to get rid of the duplicate characters and use that first, mainly because the design is more "mainstream" than v9.
I did stumble across a KLM (IIRC ... Microsoft's layout file) to xkb conversion utility which admitted to being an "early" release and limited. Will see if I can wangle a KLA2xkb script to produce something reasonable. Probably only on weekend though.

I'm not really happy with these python-based solutions... they should be compiled drivers. Unless it's just my system that failed to compile the python... saw lots of error messages about doing the byte-compiling but had no idea what they meant.