Skip soft hyphen when reading token

The soft hyphen should only appear if a word is broken at the hyphen
position. Filter it out.

Adjust the entity test files to reflect the new behaviour.

This fixes Issue #224 and Debian Bug #830173.

Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830173
Bug-Debian: https://github.com/tats/w3m/issues/224

tests/allentity.expected

@@ -1726,7 +1726,7 @@ CounterClockwiseContourIntegral;
 ш                           0x448          ш      ш                        ш
 ∣                       0x2223         ∣      ∣                        ∣
 ∥                  0x2225         ∥      ∥                        ∥
-­                            0xAD                                           ­
+­                            0xAD
 σ                          0x3C3          σ      σ                        σ
 ς                         0x3C2          ς      ς                        ς
 ς                         0x3C2          ς      ς                        ς
@@ -2125,7 +2125,7 @@ CounterClockwiseContourIntegral;
 &raquo                           0xBB           »      &raquo                   »
 &reg                             0xAE           ®      &reg                     ®
 &sect                            0xA7           §      &sect                    §
-&shy                             0xAD                  &shy                     ­
+&shy                             0xAD                  &shy
 &sup1                            0xB9           ¹      &sup1                    ¹
 &sup2                            0xB2           ²      &sup2                    ²
 &sup3                            0xB3           ³      &sup3                    ³