Do not backslash-escape non-ASCII UTF-8 bytes on POSIX

Previously, kShellValid marked bytes >= 0x80 as invalid (0), causing
EscapeStringToString_PosixNinjaFork to insert an ASCII backslash (0x5c)
before each individual byte of multi-byte UTF-8 sequences. This broke
multi-byte UTF-8 characters apart and produced invalid UTF-8 in
generated .ninja files.

Mark bytes 0x80-0xff as valid in kShellValid since non-ASCII bytes are
not special characters in POSIX shells.

Bug: 579608206
Change-Id: I117a6f377b72b6b6646af89bbb4db0d4fd857846
Reviewed-on: https://gn-review.googlesource.com/c/gn/+/28280
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Reviewed-by: Matt Stark <msta@google.com>
Commit-Queue: Matt Stark <msta@google.com>

src/gn/escape.cc

diff --git a/src/gn/escape.cc b/src/gn/escape.cc
index 8d902d4..a3c139a 100644
--- a/src/gn/escape.cc
+++ b/src/gn/escape.cc
@@ -39,7 +39,16 @@
 //  `  a  b  c  d  e  f  g  h  i  j  k  l  m  n  o
     0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 //  p  q  r  s  t  u  v  w  x  y  z  {  |  }  ~
-    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0};
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0,
+// 80-ff: non-ASCII bytes (e.g. UTF-8 multi-byte sequences) are valid
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
 // clang-format on
 
 constexpr std::array<char, 256> PosixEscapesTo(const char (&entries)[256]) {

src/gn/escape_unittest.cc

diff --git a/src/gn/escape_unittest.cc b/src/gn/escape_unittest.cc
index 62fd2e2..5d501d6 100644
--- a/src/gn/escape_unittest.cc
+++ b/src/gn/escape_unittest.cc
@@ -66,6 +66,11 @@
   // Curly braces must be escaped to avoid brace expansion on systems using
   // bash as default shell..
   EXPECT_EQ("\\{a,b\\}\\{c,d\\}", EscapeString("{a,b}{c,d}", opts, nullptr));
+
+  // Non-ASCII UTF-8 bytes (>= 0x80) are not special to POSIX shells and must
+  // not be backslash-escaped byte-by-byte.
+  EXPECT_EQ("föo\\$ bär\\$ \\<あ\\>",
+            EscapeString("föo bär <あ>", opts, nullptr));
 }
 
 TEST(Escape, NinjaPreformatted) {