Add consvar for TEMPFILE encoding

Alternate approach to avoid decoding problems: don't try to guess how
to encode the TEMPFILE.  Leave it utf-8 unless TEMPFILEENCODING is set.

Add comment to TEMPFILEDIR doc on the topic as well.

Signed-off-by: Mats Wichmann <mats@linux.com>

Author: mwichmann

CHANGES.txt

@@ -83,9 +83,9 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
       Windows 11 and GitHub windows-2025.
 
   From William Deegan:
-    - Fix Issue #4746. TEMPFILE's are written with utf-8 encoding, this doesn't work
-      for all situations. Change to use locale.getpreferredencoding(False), which should
-      yield the encoding for the system.
+    - Fix Issue #4746. TEMPFILE's are written with utf-8 encoding, In case
+      of decoding errors, TEMPFILEENCODING can now be specified to give
+      more explicit instructions to SCons.
 
   From Edward Peek:
     - Fix the variant dir component being missing from generated source file

RELEASE.txt

@@ -76,9 +76,13 @@ FIXES
 - Fix a test problem on Windows where UNC tests failed due to incorrect path
   munging if a non-default %TEMP% was defined (as in moving to a Dev Drive).
 
-- Fix Issue #4746. TEMPFILE's are written with utf-8 encoding, this doesn't work
-  for all situations. Change to use locale.getpreferredencoding(False), which should
-  yield the encoding for the system.
+- Fix Issue #4746. The TEMPFILE is written in utf-8 encoding by default.
+  If the tempfile contents cannot be decoded by the command the
+  tempfile is passed to, (new) TEMPPFILEENCODING can be used to speficy a
+  different encoding to use. On Windows, the username may be a cause of this,
+  as the default path for temporary files includes the username. Setting
+  (existing) TEMPFILEDIR may also help in this case.
+
 
 IMPROVEMENTS
 ------------

SCons/Platform/Platform.xml

@@ -395,9 +395,18 @@ Note this value is used literally and not expanded by the subst logic.
 <summary>
 <para>
 The directory to create the long-lines temporary file in.
-If unset, some suitable default should be chosen.
-The default tempfile object lets the &Python;
-<systemitem>tempfile</systemitem> module choose.
+If unset, the &Python;
+<systemitem>tempfile</systemitem> module chooses the directory
+based on the <envar>TMPDIR</envar>,
+<envar>TEMP</envar>
+or <envar>TMP</envar> environment variables.
+If the default path causes processing errors,
+set &cv-TEMPFILEDIR; to a safer alternative.
+For example, on Windows,
+the default temporary file path contains the username.
+If the username contains non-7-bit-ASCII characters,
+there may decoding errors opening the path to the temporary file.
+See also &cv-link-TEMPFILEENCODING;.
 </para>
 </summary>
 </cvar>
@@ -411,8 +420,8 @@ If you need to apply extra operations on a command argument
 (to fix Windows slashes, normalize paths, etc.)
 before writing to the temporary file,
 you can set the &cv-TEMPFILEARGESCFUNC; variable to a custom function.
-Such a function takes a single string argument and returns
-a new string with any modifications applied.
+The function must accept a single string argument and
+and return a new string with any modifications applied.
 Example:
 </para>
 
@@ -435,4 +444,21 @@ env["TEMPFILEARGESCFUNC"] = tempfile_arg_esc_func
 </summary>
 </cvar>
 
+<cvar name="TEMPFILEENCODING">
+<summary>
+<para>
+By default, the long-lines temporary file (aka "response file") created by the
+&cv-link-env-TEMPFILE; function will be encoded in the &Python;
+default encoding, UTF-8.
+If the external command which reads the response file
+encounters decoding errors
+(usually, if that command depends on Windows legacy code pages,
+and a pathname in the response file
+or the response file path itself cannot
+be represented in the 7-bit ASCII characer set),
+set this variable to the appropriate codec.
+See also &cv-link-TEMPFILEDIR;.
+</para>
+<para><emphasis>New in version NEXT_RELEASE</emphasis></para>
+
 </sconsdoc>

SCons/Platform/__init__.py

@@ -47,11 +47,13 @@
 import os
 import sys
 import tempfile
-import locale
 
+import SCons.Action
 import SCons.Errors
+import SCons.Platform
 import SCons.Subst
 import SCons.Tool
+import SCons.Util
 
 
 def platform_default():
@@ -257,7 +259,8 @@ def __call__(self, target, source, env, for_signature):
         else:
             tempfile_dir = None
 
-        fd, tmp = tempfile.mkstemp(suffix, dir=tempfile_dir, text=True)
+        # default is binary - encode the tempfile contents later
+        fd, tmp = tempfile.mkstemp(suffix, dir=tempfile_dir)
         native_tmp = SCons.Util.get_native_path(tmp)
 
         # arrange for cleanup on exit:
@@ -280,7 +283,8 @@ def tmpfile_cleanup(file) -> None:
         tempfile_esc_func = env.get('TEMPFILEARGESCFUNC', SCons.Subst.quote_spaces)
         args = [tempfile_esc_func(arg) for arg in cmd[1:]]
         join_char = env.get('TEMPFILEARGJOIN', ' ')
-        os.write(fd, bytearray(join_char.join(args) + "\n", encoding=locale.getpreferredencoding(False)))
+        encoding = env.get('TEMPFILEENCODING', 'utf-8')
+        os.write(fd, bytes(join_char.join(args) + "\n", encoding=encoding))
         os.close(fd)
 
         # XXX Using the SCons.Action.print_actions value directly