NTSTATUS

.NET Preprocessor Visual Studio 2010 Add-In Update

2011-09-07T23:23:00.001+08:00

NTSTATUS: STATUS_DOTNET_MACRO

Here’s a new version of VSDotNetPP: Version 0.2.0.0. It can be downloaded here.

Major updates and additions in this version:

Macros in open documents are now expanded in the file itself, therefore eliminating the previous message saying that the file has been updated outside the editor. This makes ‘Restore Macros In Source’ practical, but still not recommended
Context-menu item to expand macros. This expands all macros in the current document.
DOTNETPP.EXE is no longer included in the install

‘Expand Macros’ in context-menu:

As before, building will automatically expand the macros. If ‘Restore Macros In Source’ is set, then the macros will be restored in the document after the build is completed.

.NET Preprocessor Update

2011-01-20T03:18:00.002+08:00

NTSTATUS: STATUS_DOTNET_MACRO

Latest versions:

DOTNETPP Version 0.1.12.0
VSDotNetPP Version 0.1.3.0

As before, they can be downloaded here.

VSDotNetPP now implements DOTNETPP and does not call DOTNETPP.EXE, but it is still included in the install anyway. This update fixes some bugs and makes some processing changes (supposedly for the better). Following are the specific feature updates.

Features Updated In The .NET Preprocessor

The setting for <MacroFilePath> now accepts a directory (path), and parses all .npp files for macro definitions. Or, you can just specify a file path (does not have to have a .npp extension).

A list of the source files processed is kept temporarily to avoid restoring files unecessarily during the cleanup process. It was an annoyance otherwise.

Other than that, some bugs have been fixed.

Features Updated In The Visual Studio Add-In

A new setting has been added: Close and re-open source files with macros during build

I added this setting because I was getting annoyed at the source files getting closed. It’s OK if you have one or two, but it’s not OK if you have more open. Also, I think it should be obvious that the main problem with the closing/opening is that it currently does not return you to the same spot in the file you were in.

I recommend the following settings in the image:

NOT restore macros in source
NOT close/re-open the source files
Cleanup expanded source

By using these settings, it’s pretty much the same as using DOTNETPP in your pre-build and post-build events. But with the way it is currently, it works the best. So to do a build, you’d have to build twice. The first build expands the macros, and you have to say YES when Visual Studio asks if you want to reload the files (source changed the Preprocessor). I just build when I want to expand the macros.

Finally

This is pretty much it for the time being. I’ve been using it and it’s been working well. I also found it was quite useful for SQL scripts as well. It managed to reduce some copy/paste/edit.

Here’s a useful macro for the ternary operator:

#define HASVAL(a) a.HasValue ? a.Value : 0

IntelliSense does not do regions:

#define REGPRV

#region Private

#endregion

Visual Studio & Expression Blend crashes when opening Silverlight XAML

2011-01-16T01:13:00.005+08:00

NTSTATUS: STATUS_FATAL_ERROR

This happened to me recently, and it was a nightmare to repair. I will document my experience as many people faced the same problems, and different solutions worked for different people.

I was using Visual Studio 2010 and Expression Blend 4, developing Silverlight 4.

I will put the important points of this post here instead of at the end to save you, the reader, time.

Determined Root Cause of Damage: Updated Silverlight runtime to the latest version.

Caveat: Do NOT install a newer version of Silverlight unless you have the matching developer runtime (if you are developing Silverlight apps). I would also point out that it was IMPOSSIBLE to find the matching Silverlight developer runtime for the latest version of Silverlight. I am extremely infuriated and disappointed.

1. The Problem As It Happened

Suddenly, whenever I opened a Silverlight XAML file, Visual Studio crashed. It does not happen to WPF XAML files. I don’t think I tried Blend at this point as I was trying to get some work done, and was trying to fix my VS.

Steps taken:

Uninstalled Silverlight 4 Toolkit
Reinstalled Silverlight 4 SDK. Did not do an uninstall as the SDK documentation said it was safe to do (unless I read it wrong)

Results:

VS still crashes when opening Silverlight XAML
Blend 4 crashes when opening Silverlight XAML. It cannot be confirmed if the steps taken caused this, or it was already damaged

2. Reinstallation

The next desperate step was to reinstall my dev tools. This did NOT fix the problem. But I will detail it for the record.

Steps taken:

Uninstalled Silverlight 4 Toolkit
Uninstalled Silverlight 4 SDK
Uninstalled Silverlight 3 Toolkit
Uninstalled Silverlight 3 SDK
Uninstalled Silverlight Runtime
Uninstalled Visual Studio 2010
Uninstalled Visual Studio 2008
Uninstalled Blend 4
Uninstalled Blend 3
Install Visual Studio 2010
Install Blend 4
Install Silverlight 4 SDK
Install Silverlight 4 Toolkit

Results:

VS still crashes when opening Silverlight XAML
Blend 4 still crashes when opening Silverlight XAML

At this point, it is safe to say that the system is irreparably damaged. I read that some folks had to reinstall their entire system. So not wanting to waste any more time, I restored my last good system image.

3. System Restore & Update

Having restored my system to the last good image, I attempted to update it. I verified that VS and Blend are working properly with Silverlight 4 XAML files. However, when I ran my Silverlight 4 app, it insisted I install a newer version of the Silverlight runtime and just refused to run. I could not understand this as I had just restored my system to an older state which actually is using an older version of Silverlight (4.0.50524.0). I figure this out later. I had no choice at the time, so…

Update Steps:

Update Silverlight runtime to version 4.0.51204.0 (Current version is 4.0.50524.0)

Results:

VS cannot debug Silverlight 4 app because the developer runtime does not match the current Silverlight version

So this is a big problem. To fix this, I tried the following:

Uninstall Silverlight runtime
Download and install the latest Silverlight 4 developer runtime that I could find

Result:

VS can now debug Silverlight 4 app again
However, VS now crashes when opening Silverlight XAML
Blend 4 also now crashes when opening Silverlight XAML

So, it looks like this was how my system got damaged in the first place. I vaguely recall installing the latest version of Silverlight when prompted some weeks or months ago, and I had not worked on a Silverlight app until now.

I restored the system AGAIN, and did NOT install the latest version of Silverlight. However, there was still that nagging problem of my app insisting I install the latest version of Silverlight! I finally figure it out:

Silverlight App Project

The param “minRuntimeVersion” of the Silverlight object in the host webpage is what causes the popup to install a newer version of Silverlight if the value is above the current version installed. So just make sure that this is no greater than your current version of Silverlight.

I believe that I had created the said Silverlight app project after installing the newer version of the Silverlight runtime, and this is what caused the popup dialog to insist on the newer version of Silverlight even after I restored my system. This would make sense as my project was not affected by the system restore.

Conclusion

Ultimately, I was not able to repair the damage. I could only restore my system back to a point where my development system was all good. The only thing I could really do was to make sure that I do not take any of the steps that would cause that kind of damage. And these steps, are those I laid out which I figured out through trial and error. I could have figured it out faster, but unfortunately, installs, uninstalls and system restores take a very, very long time. In all, it was about 18 hours wasted because nothing new was achieved… It was all just to get back to square one.

The .NET Preprocessor Visual Studio Add-In

2011-01-05T16:55:00.003+08:00

NTSTATUS: STATUS_DOTNET_MACRO

So this is the obvious next best step: a Visual Studio Add-In for DOTNETPP to integrate it (somewhat) with VS. The biggest problem it would fix is ‘the catch’ mentioned in earlier posts regarding open source files that have macros.

The Add-In VSDotNetPP can be downloaded here. The download includes the latest version of DOTNETPP (Version 0.1.10.0).

Do note that this Add-In was built with Visual Studio 2010 for Visual Studio 2010. I do not believe it will work with an older version. The Setup created isn’t the greatest, it’s as basic as installing the files to a directory, and is uninstall-able. I’ll try to create a more comprehensive setup package later, but for now, there are some manual steps.

Registering the add-in

The default install directory is: “C:\Program Files\DOTNETPP\.NET Preprocessor Visual Studio Add-In”. You need to run the batch file Deploy-AddIn.bat to register the add-in with VS. This batch file copies the add-in file VSDotNetPP-Deploy.AddIn to the directory "%HOMEPATH%\Documents\Visual Studio 2010\Addins" which is one of the paths that VS2010 searches for add-ins. As I am using Windows 7, I am not sure if this path is valid for Windows Vista or XP.

You can find the paths that VS2010 searches for add-ins in your options settings by going to Tools->Options->Environment->Add-in/Macros Security. You may place the add-in file in any of those directories. This is the information from MSDN.

After registering the add-in, start VS2010, and you should see a new item in the ‘Tools’ menu called ‘.NET Preprocessor’.

To remove the add-in from VS, just remove the add-in file from the path you put it in and restart VS.

How the Add-In works

The add-in basically does what was suggested in previous posts: it closes the relevant open source files prior to running DOTNETPP… And, it opens them back after the build completes! So with this, you will not be bugged with the dialog asking if you want to reload the file as it has been modified outside the editor and fail the build. With this, you can choose to retain the macros in your source with much less inconvenience. However, I’d still not recommend that. I know it’s an inconvenience, but I don’t know how to set the view of the open file back to where it was before it was closed yet. When I figure it out, I will add that feature.

The add-in will call DOTNETPP for pre-build and post-build, so you will no longer have to add the calls yourself into the pre-build and post-build events in your project settings.

Settings Dialog

There is a settings dialog when you click on the ‘.NET Preprocessor’ menu item in the ‘Tools’ menu. This is basically for editing the DOTNETPP config file. Enter the full path of the config file into the settings dialog, and it will load all the settings. If the config file path is invalid, or it is blank, then DOTNETPP will not run during the build. This is how you can ‘turn it off’.

The rest of the settings have already been explained in the previous post about it.

Some Notes

OK, writing the add-in was really tricky. It was not straight forward at all, and I think I caught most of the ‘catches’, and I’ve been using it, and it works fine for the most part. However, if it is not behaving right, like changed settings don’t seem to be taking effect, I can only suggesting restarting Visual Studio. I know, it’s dumb, but that’s what I did and it was fine. However, in the latest testing, this problem hasn’t turned up, so it’s possible I fixed it. As I said, building add-ins for VS is not straight forward eventhough it looks like it is.

Before I forget, the add-in creates an XML file called VSDotNetPP.xml in your solution path. It is used to keep track of the DOTNETPP config file path.

Send me email at tim@bugslayersrus.com to comment, report bugs, or chat. Or, you could leave a comment here. Thanks.

The .NET Preprocessor - Part 4

2011-01-03T21:13:00.008+08:00

NTSTATUS: STATUS_DOTNET_MACRO

OK, so now let’s see what macros would be useful… But before we actually get to that, I think we should look at the motivations and the sensibilities behind it. For those who are or were not C/C++ programmers, this would be relevant. Also, the article on the C Preprocessor in Wikipedia is a good place to start.

One of the first points, I believe, is that the macros should help you reduce or simplify your work/effort. Apart from general term replacement as in declaring what would be const values and such, is that the macros can help you expand arguments into pieces of code. So the common sense here would be that if it takes more effort to use the macros than actually writing the code, then it’s probably a poor choice in using that macro.

The only real guidelines I can recommend are the following:

If you start to get annoyed about having to write some code because you’ve done it so many times and want some magic shortcut, then it’s a good candidate to be a macro.
If you are writing a piece of code that is repetitive, but a term or two changes just a little bit here and there, then it’s a good candidate to be a macro.
If calling the macro takes more effort than writing the code itself, then it might not be a good idea.

Because IntelliSense is so good now, some code don’t need to be macros as keystrokes have been reduced dramatically for languages like C# and VB.NET. Not so lucky if you’re writing C/C++. IntelliSense still (until today) behaves like a retarded potato that is two bricks short of a load. I just don’t get it. Do they think that Visual C/C++ don’t need IntelliSense to pop-up as soon as you type and make clever suggestions and auto-complete for you? OK, now that I’ve gotten that out of the way, you get the point.

If a macro results in more keystrokes, then it might not be that great an idea. Also, do take note that the macro (keystroke) recorder is a brilliant tool which I love, that is perfect for adjusting text with a pattern or structure. It’s an absolute gem of Visual Studio. However, that isn’t really practical in actually writing code.

Here are some macros that you might useful for C#. I use some of them:

As for the names of the macros (the identifier), those above are basically cryptic. This is because I’m keeping them short, for if not, it’ not be practical as I’d have to type a long name, which might make me punch more keys than if I did not use the macro in the first place. So it’s really up to you how you choose to name it. Refer to guideline no.3 above.

So really, you can do quite a lot with macro expansion, and as cliché’d as it sounds, it’s really up to your imagination. For example, the macro PROPVM(p) above is something that would be useful for implementing MVVM, and you’re not using any magic framework. Even a basic Property like that though simple to write, becomes tedious when you need to do a lot of it. Yes, you can copy/paste and modify, you but it does get tedious, doesn’t it? The macro would have reduced work significantly.

The .NET Preprocessor - Part 3

2011-01-03T18:40:00.004+08:00

NTSTATUS: STATUS_DOTNET_MACRO

Now let’s go through the config file that DOTNETPP expects. It’s an XML file, and doesn’t expect any special name, and I called mine dotnetpp.xml. Here are the settings:

<?xml version="1.0" encoding="utf-8" ?>

<MacroFilePath>.\MyMacros.npp</MacroFilePath>

<SourceBasePath>C:\MyProject</SourceBasePath>

<RestoreMacrosInSource>false</RestoreMacrosInSource>

<CleanupOriginalSources>false</CleanupOriginalSources>

<ClenupExpandedSources>false</ClenupExpandedSources>

</Diagnostics>

</Settings>

Following are the meanings of each setting item:

Language: [C#/VB]

Supports only ‘C#’ and ‘VB’ where it will look for files of their respective extensions.

Extension: source file extensions including the dot (.cs, .vb)

OK, so it doesn’t only specifically support only .NET languages like C#/VB.NET. It will parse files of any extension you specify here. And this setting will override the <Language> setting.

MacroFilePath: Full path to the macro file

No specific file name, but will expect .npp as an extension. Currently, this setting expects the full path to a file, but does not harp on the file name or extension. However, the next feature upgrade will allow you to put a path (directory) in this setting, and it will look for all .npp files to parse for marco definitions.

SourceBasePath: The base path of the source files which require macro expansion

Usually would either be the path of your solution, or project.

RecurseSubDirectories: [true/false]

Recurse the subdirectories of <SourceBasePath> in search of source files

RestoreMacrosInSource: [true/false]

Restore the macros in the source files after expansion. Refer to ‘The Catch’ in Part 2. As mentioned prior, might be a better way to NOT restore macros and have them ‘permanently’ expanded into actual source.

CleanupOriginalSources: [true/false]

Delete all saved copies of source files before the expansion

ClenupExpandedSources: [true/false]

Delete all copies of the expanded source files. They have the extension .nmx. There was some challenge in finding an extension that was not already used and had an associated application.

stdout: [true/false]

Output messages to stdout.

debug: [true/false]

Output messages to the Debugger. Can be viewed with DebugView from Sysinternals.

The .NET Preprocessor - Part 2

2010-12-31T02:25:00.008+08:00

NTSTATUS: STATUS_DOTNET_MACRO

The .NET Pre-Processor may be downloaded here.

The first debate was which language it should be written in. The candidates were C/C++, C# and Python. I really wanted to write it in C/C++, but the conveniences of C# and the .NET platform was just scoring too many points. No, I will not write managed C++ in Visual C++. I do not like the syntax. As for Python, the code would probably have been shorter, but it’s not a common install, and not as commonly used, I think. Related to that, Python got a boost when an actual Python came to my house a week ago. No, seriously. It’s the second one to come by in 6 months. I told my good friend Walter (Developer Security MVP) that it might have been a sign. But alas, it was C#.

DOTNETPP is a command-line tool that is largely designed to work with Visual Studio’s pre and post-build facilities. The settings are expected to be in an XML file, a sample of which is included in the download. Following is the usage:

Usage: DOTNETPP /p:"<config file path>" [/process|/cleanup]

/process: Process macro expansion

/cleanup: Cleanup expanded macro files

The config file path is specified by the /p: parameter. /process and /cleanup are actually mutually exclusive. If they both appear, /process will take priority. So basically, you will use /process in your Visual Studio .NET Project’s pre-build event, and /cleanup in your post-build event. Of course you can also manually run it to process and cleanup.

Macro Definition File

The macro definition file is basically a text file where you define your C Preprocessor macros. I will post some examples subsequent posts. DOTNETPP supports the following macro/preprocessor features:

- Non-parameterized macros

- Parameterized macros

- Token concatenation (##)

- Stringizing Operator (#)

- C++ style comments (//)

Since the macro definitions are in its own file, the line-continuation backslash ‘\’ is not required at the end of each line of the token list.

Source Files

In the source files which you want to use macros and have DOTNETPP process them, include the following syntax in said source file:

#include DOTNETPP

Since this syntax is not processed, it must be commented or excluded from compilation for the build to succeed. In the case of C#, you may use the following:

//#include DOTNETPP

#if false

#include DOTNETPP

#endif

The Catch

There is a catch. This is because this tool is NOT integrated into Visual Studio. This is what the problem is… the way DOTNETPP works is that it (obviously) expands the macros, and then writes it back into the source file. So the catch is, if any of the source files that are processed by DOTNETPP are open when the build runs (with DOTNETPP running in the pre/post-build events), the build WILL FAIL.

Why? This is because Visual Studio will ask if you want to reload the file as it has been edited outside of the source editor. The compilation would have started way before you’re able to say you want to reload the files. So if you reload and save the changed files, you will LOSE all your macros, and have the expanded code instead. This would be equivalent to setting <RestoreMacrosInSource> to FALSE in the config file.

Here’s the scenario:

1. Build project/solution

2. Build fails

3. Visual Studio asks if you want to reload the files

4. Reload the files and save

5. Build project/solution (this time it will succeed assuming you did not screw up the macro definitions)

This is a clearly a serious inconvenience to have to close all the source files with the macros before doing a build every time. This is a major con. Well, it’s not part of Visual Studio, tough. So maybe there is a way to work around this if the tool is integrated into Visual Studio. That would be something to look into for the future.

However, having thought about it, this problem probably isn’t necessarily a bad thing. I probably don’t actually want the macros to remain in the source as it can be cryptic. So, as a potentially fortunate consequence of the way Visual Studio works, DOTNETPP could work as a one-time-no-return macro expander. After all, one of the config settings is <RestoreMacrosInSource>true/false</RestoreMacrosInSource>. OK, so maybe this is a less ‘evil’ way. Expand the macros so they become the source code, and no more macros in the source.

Next I will write about the config file, but the elements should be quite obvious.

Here’s one of the main reasons I started this project:

#define FOR(a) for (int i = 0; i < a.Count; i++)

The .NET Preprocessor - Part 1

2010-12-30T22:47:00.002+08:00

NTSTATUS: STATUS_DOTNET_MACRO

This is my only technical blog, so I’ll just put it here for now…

My favourite programming language is still C. One of the things I liked a lot about it is the C Preprocessor. There are people who love it, and then there are those who hate it. Some say that it is evil. Like they say, with C, you shoot yourself in the foot. I guess the C Preprocessor (CPP) is the same. It can be used for good, or evil. If used appropriately, boy, can it save you a load of tedious coding/keystrokes. If you screw it up, you shoot yourself in the foot.

So nowadays, I don’t use C/C++ as much as there is not all that much need for it, and better languages/platforms for certain jobs… Like web apps, and general Windows apps… Yes, that makes up most of the needs nowadays. I’ve always been Windows kinda guy, so naturally I’m using C# on .NET. Man, C# is sweet, and the .NET libraries are super sweet. OK, so at least C# has some basic compiler-directives, like symbol definition and conditional compilation. From a C background, that is very welcome indeed.

However, there is NO macro expansion!!! [Insert expletive here]. If you’ve used the CPP, you’ll soon start to miss it. I swear, man, I am so sick of typing tedious repetitive code. If you are one of those who would say to substitute the macros with functions, then forget it because I don’t want to create more overhead where it can be avoided. Besides, it just isn’t the same concept or in the same spirit.
So about a month ago, I started moaning about this again because I am so, so, so, did I say I was sooo sick of writing for-loops (for instance)? So I kindly asked someone (anyone) to add the CPP/Precompiler to C# on Facebook. OK, I know it was not included in C# for reasons unknown to me. But I believe Hejlsberg and co. had good reasons. But I know what I am doing, and I WANT IT!!! So how now brown cow?

So, having toyed with the idea for a couple of weeks, I spent another week or two actually thinking it out in my head how to build it, and which features it would include. Of course, I thought about how it could work with Visual Studio. Not surprisingly, after a few weeks of laying it all out in my head, I finally took some ‘time off’, sat down, and went about building it. Surprisingly, it took me just over half a day to finish the first complete app. So next time your boss catches you seemingly doing nothing, just tell ‘em that you’re laying it all out in your head first so the execution will be quicker and smoother.

Anyway, I will continue to write about how it works in subsequent posts and put the links up. Oh, I beg your pardon, I haven’t introduced it. I call it the ‘.NET Pre-Processor’, and it is DOTNETPP.EXE.

Dissassemble Me

2010-09-07T16:59:00.006+08:00

NTSTATUS: STATUS_OBJECT_DUMPED

A must have tool is PEBrowse Pro by SmidgeonSoft written by the great Russell Osterlund. Use it to disassemble Win32/Win64 and .NET (assemblies) PE binaries (EXE/DLL).

I have been using this great tool as a lightweight disassembler for a decade. It is updated fairly often improving features. It's really nice tool to use to see a Win32 binary's imports and exports. Much nicer than using DUMPBIN, unless you prefer a command line app. They also have a debugger as well that works on Win32 apps and .NET apps.

Process Session

2010-03-01T00:33:00.012+08:00

NTSTATUS: STATUS_JAM_SESSION

I think there are a fair number of people who are not aware of the concept of 'sessions' in Windows, and some are aware of it, but only that there is such a thing. We're going to go through the basics of it, and see how it might be relevant.

A session in Windows NT is basically a user login session. It is managed by the Session Manager Subsystem process (SMSS.EXE). This includes Terminal Services/Remote Desktop sessions. Windows NT servers allow multiple concurrent Terminal Services login sessions whereas the client versions Windows NT does not.

If I recall correctly, in the initial version of Windows XP, you could be logged in to Windows, and have a Remote Desktop session active with another user being logged in. So, you could actually have two simultaneous user sessions. Then it came to pass, in, I think it was SP2 that Terminal Services was changed such that you could not run a Remote Desktop session without having the current user logged off. I think the story was that the change was made because the way Terminal Services was working was in violation of the EULA which allowed only one user of that copy of Windows XP at any one time.

Now, moving along, sessions weren't terribly relevant prior to Windows Vista with the exception of the case of Windows XP prior to the Service Pack with changed Terminal Services. This is because prior to Vista, you could not actually have more than one user logged in to Windows at the same time. You'd remember this because if you wanted to login as another user, you had to logout of your current session. So in that sense, it wasn't actually relevant for your apps to be concerned about which session it was running in.

However, since Vista, you could login multiple users in your machine which I really appreciated because you didn't have to lose your session. I hated having to have my session restarted and to get things back to the way they were, and it took an amount of time I wasn't interested in waiting on.

So now, in Vista and Windows 7, the session in which your process runs becomes relevant. There are two basic scenarios here:

There should only be one instance of your app running in Windows
There should be only one instance of your app running in each user session

In the first case, it is pretty straightforward. You would enumerate all processes and see if there is already in instance running.

In the second scenario, it is commonly the case where an instance of your app is required to deal with one (instance of a) user specifically. Therefore, if there was another user logged in, you needed another instance of your app to run in that user's context as well. In most cases, this would be a UI app. Also, it is an app where you only want only a single instance running (per user session). So in the old days before multiple concurrent user sessions, you would just check to see if the app is already running, and not start up a second process. If that was done now, you will end up with only one instance of the app running regardless of the number of user sessions.

So now we get to the gist of it, which is: ensuring only one instance of your app runs per user session.

To do this, we need to enumerate all instances of the app in question, and check the session in which it belongs to. As an aside, you can use Process Explorer from Sysinternals to look at the process tree to see all the session running, and their associated processes. And obviously, you are not able to access the processes that do not belong to your current session.

The Session ID of a process can be obtained from the Process Environment Block (PEB) Win32 structure. This one of those magic data structures that are largely undocumented, and in most instances, for good reason. Following is the struct from the Windows SDK version 7.0:

typedef struct _PEB {
BYTE Reserved1[2];
BYTE BeingDebugged;
BYTE Reserved2[1];
PVOID Reserved3[2];
PPEB_LDR_DATA Ldr;
PRTL_USER_PROCESS_PARAMETERS ProcessParameters;
BYTE Reserved4[104];
PVOID Reserved5[52];
PPS_POST_PROCESS_INIT_ROUTINE PostProcessInitRoutine;
BYTE Reserved6[128];
PVOID Reserved7[1];
ULONG SessionId;
} PEB, *PPEB;

You can see the SessionId field in the struct, which is what we're interested in. Another field of potential interest is the Ldr field which contains a doubly-linked list of the modules loaded by the process.

The information of this struct is obtained via the Win32 API function:

NTSTATUS WINAPI NtQueryInformationProcess(
__in       HANDLE ProcessHandle,
__in       PROCESSINFOCLASS ProcessInformationClass,
__out      PVOID ProcessInformation,
__in       ULONG ProcessInformationLength,
__out_opt  PULONG ReturnLength
);

Pass in ProcessBasicInformation in the ProcessInformationClass parameter, and the function will return a pointer to a PEB structure.

So there you have it, the session in which the process is running.

Happy Holidays!

2009-12-24T14:17:00.003+08:00

NTSTATUS: STATUS_W00T

Here's wishing everyone a Merry Christmas, and a Happy New Year! :)

"You cannot be serious!"

2009-12-06T12:40:00.004+08:00

NTSTATUS: STATUS_IS_NOT_WIN32

Following up from my previous post...

I found a novel way to crash WPF (.net 3.5 SP1). Here's how to do it:

Create a WPF DLL:
1. Create WPF User Control Library
2. Create a User Control, say, MyUserControl
3. Create another User Control, say, MySubUserControl
4. Add MySubUserControl (as a project control) into MyUserControl (in the XAML)

So now you have a user control that has a user control in it. Next:

Create an Application (I created a .net WinForm app):
1. Create a WinForm Application
2. Load the DLL created, via Assembly.Load(byte[] rawAssembly)
2.a. Create an instance of the type MyUserControl
2.b. Add the instance of MyUserControl in a container (I just put it into an ElementHost)
3. Repeat setps 2, 2.a. and 2.b.

Basically, in the WinForm app, you just need to run your code that does steps 2, 2.a. and 2.b. twice. I just execute that piece of code via a button's event, so I can run it by pressing the button twice.

On the second execution of the code, it crashes. The exception I got is:

"Cannot create instance of 'MySubUserControl' defined in assembly 'WpfControlLibrary1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. Exception has been thrown by the target of an invocation. Error at object 'System.Windows.Controls.Canvas' in markup file 'WpfControlLibrary1;component/MyUserControl.xaml' Line 9 Position 6."

So, long story short, it seems that you can't multi-load, and re-instantiate a WPF user control, that has a user control inside it! You can imagine the shock my colleague was in for when I broke the news.

I was doing this basically to implement plug-ins. Tough luck, apparently... Who would expect this to happen? Certainly not me. This can't be right. It had me doing a Johnny Mac.

"I thought you were still there..."

2009-10-24T02:06:00.027+08:00

NTSTATUS: STATUS_IS_NOT_WIN32

This has to do with the .net CLR loading assemblies dynamically. Sure, it's not a Win32 thing, but it is related when I went about trying to achieve what I normally would in native Win32 apps.

The story begins quite simply, in implementing basic plug-in type architectures. When one did it in regular (native) Win32 apps with C/C++, you'd just use LoadLibrary() and FreeLibrary() to load and unload the relevant DLL's that would function as plug-ins.

Trying to achieve the same in .net, the methods from the Assembly class LoadFrom() and LoadFile() can be used to load an assembly at runtime. However, what's obviously missing is any method to unload those assemblies. After a little searching, we come across this. Apparently, there are plenty of reasons not to. So, this clearly begs the question, why it can be done in native Win32, and not in the .net CLR. From what I can tell, it's basically a case of trying to stop people from shooting themselves in the foot. In my case, I know exactly what I'm doing, and it's exactly what I intend to do. But too bad, everyone gets the same treatment.

It seems the closest is to read the entire assembly binary into memory, and use Assembly.Load(). Effectively, this gets the job done. So, problem solved. However, it wasn't so for me. The problem laid in WPF. This 'bug' is related to user controls and assembly loading, and I will write about that in a subsequent post.

So anyway, I wanted to see what would happen if I manually unloaded a .net assembly. I did the obvious thing having the assembly's path:

IntPtr hModule = Win32.GetModuleHandleW(assemblyPath);

if (hModule != null)
{
if (Win32.FreeLibrary(hModule))
{
Debug.WriteLine("DLL released...");
}
else
{
// failed
Debug.WriteLine("Failed to release DLL...");
}
}

Win32 in the above code is a class in which I declared the Win32 API functions. In any case, the assembly is unloaded from the process, piece of cake. So, the question now is, does anybody know about it? Obviously, the answer is NO. I suppose it is not reasonable for the .net CLR to check if the assembly is still there anytime it wanted anything from it. Heck, I wouldn't do that either since I did not let anybody else but myself do it.

Calling the code to load the assembly again does not result in the assembly being loaded again. Simply a matter of replacing the assembly with one with different content/code, and loading it again. As far as the CLR is concerned, it's already loaded. Fair enough. So the next time you called anything in the assembly, BOOM! It crashes as expected.

Oh, do note that there would still be a handle to the assembly (the DLL) in the process. It is no easy task to find the handle and close it, so I wasn't bothered as it wouldn't make any difference.

Freeing the .net DLL manually was really just a case of 'let's see what happens'. Not recommended for .net assemblies, but it's just regular for native Windows Win32 DLL's if you're working with them.

"Who's your daddy, and what does he do?"

2009-10-08T21:50:00.007+08:00

NTSTATUS: STATUS_NO_SUCH_PROCESS

Recently, I had stumbled upon some bizarre behaviour relating to Windows processes. Here's how it went...

I noticed that a process (of an app I wrote) had been running for a couple of minutes and still had not completed. It usually takes only a matter of seconds to complete (this is in Windows Vista). So I killed it (with Sysinternal's Process Explorer), and ran it again. Same behaviour...

Before going further, the basic operation of this app is to run some checks on processes in certain branches of the system process tree. For example, to only perform a task on processes that are not services.

To dig into the problem, I proceeded to rebuild the app with debug output, and see what it says with Sysinternal's DebugView. According to the output, it appears that the process doesn't get past going through the processes in the system.

Next thing I did was to run the app in the debugger and see what happens. Well, it just keeps going, and going. Trying my luck, I hit the 'pause' button to break the process. Happily, it actually stopped in my code. It stopped in a piece of code that traverses up the process tree from any particular process. This is pretty straightforward:

I had already built a map of all the processes in the system. I got the processes via ToolHelp32. So the basic idea is (simplified):

For a process:

Stop if this is an ancestor I am looking for
Get the parent process of currently examined process
If there are no more processes, stop
Loop back to 1.

So basically, it just goes up a process's ancestry, to see if it can find a specific process. The problem that was occurring, was that it somehow ended up being an infinite loop! And how can that happen?!?

Well, here's how it happened... Two processes have each other as parents! In the instance of the problem I was having, this was how it went:

devenv.exe[PID:3824] has parent [PID:3364]
explorer.exe[PID:3364] has parent [PID:3824]

"This can't be right?!?", I thought. Taking a look at explorer.exe in Process Explorer, it did show that devenv.exe was its parent, which I know is wrong because I did not use devenv.exe to start the shell. It was the other way round, as is normally. However, Process Explorer, or rather, Mark Russinovich (We're not worthy!!!) knows it's wrong, and indicates that the parent process is a 'Non-existent Process'.

So, I reworked the code to account for such a situation, and it works nominally again. In retrospect, I think it's fair that I did not write the code to expect such a situation, or to even think of it.

It was late and I was tired, so I turned in right after that. Later, I realized I should have taken some screenshots as I still have no idea how that whole fiasco happened...

Process Creation API Functions Path (XP)

2009-08-03T19:42:00.012+08:00

NTSTATUS: STATUS_CREATE_PROCESS_NT51

Following up on the previous post on the Win32 API's process creation functions in Windows Vista SP1, now let's take a look at how it was in Windows XP SP2.

Following are the process creation functions available in the Win32 API:

kernel32.dll:

CreateProcessA()
CreateProcessW()
CreateProcessInternalA()
CreateProcessInternalW()
CreateProcessInternalWSecure()

advapi32.dll:

CreateProcessAsUserA()
CreateProcessAsUserW()
CreateProcessWithLogonW()
CreateProcessAsUserSecure()

Of note are CreateProcessInternalWSecure() and CreateProcessAsUserSecure(). These two functions do absolutely nothing. They just return zero. It's also worth noting that these two functions do not appear in Windows Vista. I suppose as a joke, it means that there is no secure way to create processes in Windows XP! GYAHAHA!!!

Below is a diagram that shows how these functions are related to each other, with the exception of CreateProcessWithLogonW():

Win32 API Process Creation Functions

The general workings of these functions are the same as they were described in the previous post for Windows Vista.

Process Creation API Functions Path (Vista)

2009-07-03T20:04:00.018+08:00

NTSTATUS: STATUS_CREATE_PROCESS_NT60

It's been a while since the last posting, so let's move right on to the current one...

We'll have a little look at the Win32 API process creation functions, and what other exported (Win32) functions they use under the hood. First, we'll take a look at how it goes for Windows Vista (SP1), and we'll look at Windows XP in a subsequent posting.

Following are the process creation functions available in the Win32 API:

kernel32.dll:

CreateProcessA()
CreateProcessW()
CreateProcessInternalA()
CreateProcessInternalW()

advapi32.dll:

CreateProcessAsUserA()
CreateProcessAsUserW()
CreateProcessWithLogonW()
CreateProcessWithTokenW()

The two 'internal' versions of CreateProcess are not documented. Also, for those who are not familiar with the Win32 API conventions, the suffix of either 'A' or 'W' as in the above functions denote whether they are the ANSI or Unicode versions of the functions.

Below is a diagram that shows how these functions are related to each other, with the exception of CreateProcessWithLogonW() and CreateProcessWithTokenW(). Those two are more complicated, and we'll leave them out for now (translated as, I haven't looked into it enough at this time):

Win32 API Process Creation Functions

So, from the diagram above, we can see that the five above functions ultimately call CreateProcessInternalW().

It is interesting to note the following of the above mentioned five functions:

CreateProcessA/W() sets up parameters and calls CreateProcessInternalA/W() respectively, nothing more.
CreateProcessAsUserA/W() sets up parameters and calls CreateProcessInternalA/W() (respectively). then sets process, thread and security information of the user [the high-level overview].
CreateProcessInternalA() sets up parameters, converts ANSI strings to Unicode strings, then calls CreateProcessInternalW().

As of all undocumented functions, whether in the Win32 API or not, the general idea is that they are undocumented because they could change between versions of the operating system. Of course, there is also the old, old conspiracy theory that they are being kept secret for advantage. Maybe a little bit of both. Fair game, if you ask me. However, I will confirm that I have seen changes in undocumented functions even between different Windows service packs. And this is includes function prototypes and behaviour.

I suppose the moral is, that if you depend on undocumented functions or behaviours, you will basically have more work to do in making sure that your applications don't blow up in different versions of the system binaries.

"Did you forget to check something?"

2009-06-03T20:53:00.016+08:00

NTSTATUS: STATUS_KABOOM

Some .net applications have been behaving strangely... Actually, practically all of them have been behaving strangely, but consistently. They crash after approximately 5 seconds upon starting up. Obviously, it must be something I did.

The reported error:

Description:
Stopped working

Problem signature:
Problem Event Name: APPCRASH
Application Name: MyApp.exe
Application Version: 1.0.0.0
Application Timestamp: 49f5e887
Fault Module Name: mscorwks.dll
Fault Module Version: 2.0.50727.1434
Fault Module Timestamp: 4757b767
Exception Code: c0000005
Exception Offset: 001496a2
OS Version: 6.0.6001.2.1.0.256.4
Locale ID: 1033

Upon debugging, the crash consistently fails at the below piece of code in mscorwks.dll (highlighted in the red box):

... where the value of the register esi is zero(0):

The Call Stack:

The modules list indicating the base address of mscorwks.dll:

Upon inspection of the disassembly, it is immediately obvious that it is a poorly written bit of code related to the 2 instructions before the faulting instruction at address 7148FF92.

The return from the (function) call at 7148FF8A is not checked for zero/NULL before being used as a pointer:

Call function:

7148FF8A call dword ptr ds:[71EB3244h]

Save the return value of the function call from the register eax into the register esi:

7148FF90 mov esi,eax

Use the value in esi as a dword pointer with offset 0Ch(0xC) and store the value at the address into eax. It appears likely to be code to get a value from the address of a struct at offset 0Ch and save it into eax:

7148FF92 mov eax,dword ptr [esi+0Ch]

The value from the struct member is then saved into a local variable via eax:

7148FF95 mov dword ptr [ebp-14h],eax

Now, there is one more detail I have not mentioned until now, and that is, the .net application(s) are being injected with a DLL as soon as its process is created. This is the 'something' I must have done, and did. This is the scenario where it is experiencing this crash. If the .net processes are being injected with a DLL after (not instantly) it has started up and running, then there isn't a problem.

So, even if the .net process is being injected with a DLL immediately after its creation, why should there be a problem? All other Win32 processes do not have this problem. If someone is going to tell me that I shouldn't be doing something like this, yadda yadda yadda, then I'd tell them to talk to the hand.

In any case, back to the crashing code, it is clearly a situation where the return value of the function being called can be a NULL (seems obvious to be a pointer) value, and it was never checked prior to being used. Although I don't know what the operational implications are to the returned pointer being NULL, I expect the .net framework to handle it properly and either return with an error gracefully, or raise a .net exception instead of crashing out.

Oh, and some nice folks at Microsoft are kind enough to take a look into this. It's very much appreciated.

New Offering From Microsoft Research

2009-05-19T19:12:00.005+08:00

NTSTATUS: STATUS_CIBAI

I swear I did not make this up... Look here: Cibai from Microsoft Research

Here's what they have to say in the abstract:

"We introduce Cibai a generic static analyzer based on abstract interpretation for the modular analysis and verification of Java classes. We present the abstract semantics and the underlying abstract domain, a combination of an aliasing analysis and octagons.

We discuss some implementation issues, and we compare Cibai with similar tools (I dunno, like 'blue bird'? :P), showing how Cibai achieves a higher level of automation and precision while having comparable performances."

I'm not sure what to think about that last statement... a "Cibai achieves a higher level of automation and precision while having comparable performances"... If that isn't speechlessly incredible in-your-dreams type of thing, I don't know what is...

I mean, heck, if Jabba The Hutt can say "Kan Ni Naa", it's all good ;)

Determining User Local Groups

2009-05-11T18:12:00.011+08:00

NTSTATUS: STATUS_WHAT_IS_MY_POWER

There might be occasions where you are developing an application where it has to take actions which depend on the permissions and privileges of the user context under which it is running. To check if a process has a specific security privilege enabled, the LSA (Local Security Authority) API functions can be used.

In this article, we will look at a simpler and more general scenario. We will make a decision based on what everybody should be well familiar with, and that is Local Groups (we'll call it LG for short). The most common ones are of course:

Administrators
Power Users
Users

So, our simple scenario is that based on the our process's user's LG membership, we will take different courses of action(s).

We will use the following Network Management Function:

NET_API_STATUS NetUserGetLocalGroups(
__in   LPCWSTR servername,
__in   LPCWSTR username,
__in   DWORD level,
__in   DWORD flags,
__out  LPBYTE *bufptr,
__in   DWORD prefmaxlen,
__out  LPDWORD entriesread,
__out  LPDWORD totalentries
);

There is an example of how to use this function in MSDN: NetuserGetLocalGroups Function

What's important here are the first two parameters. The servername would be the local computer name, or localhost can also be used in its place. The username is of course the user whose LG membership we're interested in.

The value for level should be 0, the only value defined for flags is LG_INCLUDE_INDIRECT. The data will then be return in the buffer as type LPLOCALGROUP_USERS_INFO_0:

typedef struct _LOCALGROUP_USERS_INFO_0 {
LPWSTR lgrui0_name;
} LOCALGROUP_USERS_INFO_0, *PLOCALGROUP_USERS_INFO_0, *LPLOCALGROUP_USERS_INFO_0;

The LG info returned are therefore the names of the groups itself as Unicode strings. So, string comparisons have to be done to check for the groups (such as) "Administrators", "Power Users" and "Users".

That is not how you say my name...

2009-04-24T16:43:00.019+08:00

NTSTATUS: STATUS_LMAO

If you're ever looking for some random, cryptic or WTF type of name to use as an identifier in your code, I've got one for you... And it is: Xobile.

However, there are some rather special uses for Xobile. It works best in C/C++, and can also be used in C#. Here are the characteristics:

It should be of a type where it can be tested for 0 (zero)
The type can be of a pointer type
The type can be of a numerical type
The special case is that it must be tested for 0 (zero)
The whole crux of this is to use the '!' operator on it

What we are specifically targeting for, is to write: !Xobile

Examples of declarations as variables:

Examples of declarations as a function:

Examples of usage:

Just FYI, 'Xobile' is not pronounced "so-buy-l", or "so-beel", or "so-bee-leh". But more importantly, how do you say "!Xobile"?

With no further delay, please allow me to introduce Russell Peters to give us the low-down on '!Xobile'... *Applause*

Handling And Re-Handling

2009-04-21T23:47:00.013+08:00

NTSTATUS: STATUS_IT_WASNT_REALLY_OBVIOUS

In the Windows system, an object that represents a system resource such as a process, thread, file or synchronization object are accessed via a handle. The application will have to obtain a handle to such objects which can then be used to manipulate or examine these objects.

Here are some common handle types in Win32:

Kernel Objects:

Process: HANDLE
Thread: HANDLE
File: HANDLE
Synchronization Objects (mutex, semaphore): HANDLE

User Objects:

Window: HWND
Menu: HMENU
Icon: HICON

GDI Objects:

Device Context: HDC
Bitmap: HBITMAP
Brush: HBRUSH

What we're going to focus on here are kernel objects, and a particular scenario where a not-so-obvious Win32 API function earns it's dollars.

Here's the scenario. You have obtained a handle to a kernel object, and for this example, it is a process handle (HANDLE). You did not create the process to which this handle is for, nor did you open the process to which this handle is for. You just got hold of it. Now, you want to get some information on this process, so let's start by trying to get the process's module file name with GetProcessImageFileName().

Boom! The function returns a failure. And the error reason is that you basically do not have the access permissions to do so. The handle must have one of the following permissions:

PROCESS_QUERY_INFORMATION
PROCESS_QUERY_LIMITED_INFORMATION

So clearly, it doesn't. And since you did not create or open that process, you also don't know which Process ID it was either that the handle came from. !@#$%. Guess we'll just have to try to get that information from the handle then. Oh, wait, we probably can't either coz we don't have the necessary permissions. How now brown cow?

Well, as it turns out, there is a particular Win32 API function I've known for ages, but had never examined it (clearly, I've not used it until not too long ago) as it seemed silly back then. The function: DuplicateHandle(). I guess I never paid attention to it because I didn't see the need for that as I could just use another HANDLE variable and assign it the same value. D-uh...

However, here's the prototype:

BOOL WINAPI DuplicateHandle(
__in   HANDLE hSourceProcessHandle,
__in   HANDLE hSourceHandle,
__in   HANDLE hTargetProcessHandle,
__out  LPHANDLE lpTargetHandle,
__in   DWORD dwDesiredAccess,
__in   BOOL bInheritHandle,
__in   DWORD dwOptions
);

So, it clearly does a whole lot more than just duplicating a handle, so to speak. The magic is in the parameter dwDesiredAccess. You can set the permissions for the new handle that is to be duplicated from the original. So, what you'd get is a new handle that accesses the same object as the original handle, but with different properties such as the access permissions. However, there is another catch. This won't work either unless the source process of the original handle has the PROCESS_DUP_HANDLE permission. So far, I don't think it's common to not have this permission.

Just for good measure, you can specify DUPLICATE_SAME_ACCESS in the dwOptions parameter, and the dwDesiredAccess parameter will be ignored.

Therefore, by duplicating the original process handle this way, you can now obtain some information from it, like what GetProcessImageFileName() can tell you.

NTSTATUS? Wazzat?

2009-04-20T20:32:00.009+08:00

NTSTATUS: STATUS_GENERAL_INFORMATION

In recent times, I've met a fair number of people in software development who work on Microsoft Windows who go "Win32 API? What's that?" I guess it is that people just don't, or can get by without needing to know what that is, let alone use it. I'm not saying that there's anything wrong with that. Just sign o' the times, I guess. Also a sign of what kind of development is being done more pervasively nowadays.

In this blog, unless specified, 'Windows' (as in Microsoft Windows) will generally refer to WindowsNT, and usually NT5.1 (Windows XP), and NT6.0 (Windows Vista) unless specified. I still have some good memories of Windows 9x, but they are few and far between.

So, for those who don't already know, the calling convention alias for the Win32 API is WINAPI, and the standard error does not have a special type definition. One more layer down, in Window's kernel-mode, where the neat NT Native API is used, the calling convention is NTAPI, and the standard return type is NTSTATUS.

And then, WINAPI and NTAPI is the stdcall calling convention of Intel's x86 architecture, and NTSTATUS is an unsigned 32-bit (unless you're into 64-bit) type.

Well, so that's that. Let's see if there will be anything randomly interesting coming up...