I'm still unhappy with get-childitem semantics

[Andrew Watt [MVP]]

I know this has been discussed before but I am still worried and
puzzled by the current semantics of get-childitem particularly when
used with the FileSystem provider.

One thing that worries me is the difference in semantics between
-Include and -Filter.

But why do we need both a -Filter and a -Include parameter at all? The
Help files are ... um ... unhelpful on that point as they are on the
exact semantics of either parameter. As is, so far as I can see, the
User Guide.

Examples of why I'm unhappy follow below.

First, I did what I expected would be a simple comparison to be sure
that -Include and -Filter gave me the same results doing a
"straightforward" search for *.txt. But it didn't. I got a different
count as in the following two commands.

PS C:\PowerShellScripts> (get-childitem * -Include *.txt).count
27
PS C:\PowerShellScripts> (get-childitem * -Filter *.txt).count
28

[I found that the discrepant result was a file named odd.txtcls when I
had attempted a pipe and, presumably added a cls in error. But why do
-Incude and -Filter treat this, admittedl unusual case differently?]

So far so puzzling but it becomes more puzzling. See the next four
commands.

PS C:\PowerShellScripts> (get-childitem -Include *.txt).count
PS C:\PowerShellScripts> (get-childitem -Filter *.txt).count
28
PS C:\PowerShellScripts> (get-childitem . -Include *.txt).count
PS C:\PowerShellScripts> (get-childitem . -Filter *.txt).count
28

Why does the -Filter count stay the same but the -Include count drops
to zero?

Another test was to see how -Filter behaved. In the following three
commands the same results were returned but in a different order when
* was used as the value for the (positional) Path parameter. Why
should the ordering of results be different?


PS C:\PowerShellScripts> get-childitem -Filter te*


Directory:
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts


Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1
d---- 07/07/2006 20:35 Test1
d---- 18/05/2006 09:42 Test2
d---- 18/05/2006 09:42 Test3


PS C:\PowerShellScripts> get-childitem . -Filter te*


Directory:
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts


Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1
d---- 07/07/2006 20:35 Test1
d---- 18/05/2006 09:42 Test2
d---- 18/05/2006 09:42 Test3


PS C:\PowerShellScripts> get-childitem * -Filter te*


Directory:
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts


Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 07/07/2006 20:35 Test1
d---- 18/05/2006 09:42 Test2
d---- 18/05/2006 09:42 Test3
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1


I then went on to compare the same three values for -Path (default, .
and *) using the -Include parameter and I got very different results
for the three commands.

Why do I only get two results for the following command? Why aren't
the folders test1, test2 and test3 displayed?

PS C:\PowerShellScripts> get-childitem * -Include te*


Directory:
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts


Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1

But, more confusing for me (at least) why do I get no results
displayed for the following two commands?

PS C:\PowerShellScripts> get-childitem -Include te*
PS C:\PowerShellScripts> get-childitem . -Include te*
PS C:\PowerShellScripts>

It seems to me that the semantics of the default path, . as a path and
* as a path are treated as the same when -Filter is used (except for
ordering of results in the second example) but the semantics of *
differs drastically from the default and . when -Include is used.

Now ... can somebody help to convince me that behaviour such as occurs
in the preceding examples makes good sense?

Thanks

Puzzled of Tunbridge Wells :)
alias
Andrew Watt MVP

[Keith Hill [MVP]]

Post by Andrew Watt [MVP]
I know this has been discussed before but I am still worried and
puzzled by the current semantics of get-childitem particularly when
used with the FileSystem provider.

You are not the only one. I posted a message on June 5th titled
"Get-ChildItem Woes" where I was getting frustrated by seemingly
inconsistent behavior and confusion over -include and -filter.

--
Keith

[Andrew Watt [MVP]]

Keith,

I see you didn't get a reply to that post, which is disappointing.
Maybe you should bug that problem since it may be unresolved. I
couldn't find a bug report about it on Connect today.

I have posted a bug today on this relating to the content of my post
of yesterday:
https://www.connect.microsoft.com/feedback/ViewFeedback.aspx?FeedbackID=159321&SiteID=99

For the first time (for me) in the PowerShell beta I have marked this
issue as a "blocking issue".

The problems are there. And, in my view, it's unacceptable to consider
releasing PowerShell version 1.0 until those problems are sorted out.
So, for me, it's definitely a "blocking issue".

For others on the newsgroup, if you understand the behaviour that I
described and illustrated in the post that started this thread please
try to explain it to me. Currently it simply doesn't add up (for me).

If others similarly don't understand these semantics I suggest you
vote on the bug above.

Since there doesn't seem to be a way when "voting" to indicate whether
or not you see this as a "blocking issue" it may be more meaningful,
if you are as concerned as I am, to post a new bug and mark it as a
"blocking issue" for you.

I hope this gets fixed and I hope we then get adequate opportunity to
test the "fix". This is fairly fundamental functionality. It needs to
"just work".

Andrew Watt MVP


On Fri, 7 Jul 2006 17:45:22 -0600, "Keith Hill [MVP]"

Post by Keith Hill [MVP]

Post by Andrew Watt [MVP]
I know this has been discussed before but I am still worried and
puzzled by the current semantics of get-childitem particularly when
used with the FileSystem provider.

You are not the only one. I posted a message on June 5th titled
"Get-ChildItem Woes" where I was getting frustrated by seemingly
inconsistent behavior and confusion over -include and -filter.

[Jeffrey Snover [MSFT]]

Sorry you didn't get an answer before. We know that we have some issues
here and have been working hard to see if we could get them addressed for
V1.

I'll make sure that get you a response.

[Andrew Watt [MVP]]

Thanks for the response, Jeffrey.

Personally I see it as essential that these bugs associated with
something as basic as get-childitem are fixed BEFORE version 1 is
released.

It doesn't only affect finding files and folders but when combined in
pipelines with copy-item, move-item remove-item it has ripples much
more widely.

A pipeline architecture is great if it makes use of a component that
works. It becomes visibly problematic across a worrying spectrum if a
key component or widely used component (I include get-childitem here)
is unpredictable in its behaviour.

I would urge you to put this on the "must fix before v1" list.

Andrew Watt MVP

On Sat, 8 Jul 2006 07:30:45 -0700, "Jeffrey Snover [MSFT]"

Post by Jeffrey Snover [MSFT]
Sorry you didn't get an answer before. We know that we have some issues
here and have been working hard to see if we could get them addressed for
V1.
I'll make sure that get you a response.
--
Jeffrey Snover [MSFT]
Windows PowerShell Architect
Microsoft Corporation
This posting is provided "AS IS" with no warranties, no confers rights.
http://blogs.msdn.com/PowerShell
http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx

[Staffan Gustafsson]

I second that.

I would call it a showstopper. I've allready had production problems related
to the strange behavior of gci.

/Staffan

Post by Andrew Watt [MVP]
Thanks for the response, Jeffrey.
Personally I see it as essential that these bugs associated with
something as basic as get-childitem are fixed BEFORE version 1 is
released.
It doesn't only affect finding files and folders but when combined in
pipelines with copy-item, move-item remove-item it has ripples much
more widely.
A pipeline architecture is great if it makes use of a component that
works. It becomes visibly problematic across a worrying spectrum if a
key component or widely used component (I include get-childitem here)
is unpredictable in its behaviour.
I would urge you to put this on the "must fix before v1" list.
Andrew Watt MVP
On Sat, 8 Jul 2006 07:30:45 -0700, "Jeffrey Snover [MSFT]"

Post by Jeffrey Snover [MSFT]
Sorry you didn't get an answer before. We know that we have some issues
here and have been working hard to see if we could get them addressed for
V1.
I'll make sure that get you a response.
--
Jeffrey Snover [MSFT]
Windows PowerShell Architect
Microsoft Corporation
This posting is provided "AS IS" with no warranties, no confers rights.
http://blogs.msdn.com/PowerShell
http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx

[Keith Hill [MVP]]

Post by Staffan Gustafsson
I second that.

I'll third it. It is just such an important cmdlet in day-to-day PoSH
usage. Does the team have any thoughts on how they would like these
parameters (-include/exclude/filter) to eventually work? My thinking is
that the -exclude parameter should cull items returned from the provider
unless the provider supports -exclude directly. However if that where the
cause then what would make -filter special? The other thing I don't get
about -include/exclude is if their argument values are globbed by the shell,
how does the shell know whether to recurse or not in doing its globbing?
Surely it doesn't peek at the cmdlet's parameters (-recurse).

--
Keith

[Lee Holmes [MSFT]]

To preface this all, here is our current help description of 'Filter'

-filter <string>
Specifies a filter in the native language of the provider. Filters are
more efficient than other parameters, be
cause the provider applies them when retrieving the objects, rather
than having Windows PowerShell filter the o
bjects after they are retrieved.

The source of most confusion is that the 'native language of the
(filesystem) provider' is nearly the same as our wildcarding syntax.

Additional comments in-line.

Post by Andrew Watt [MVP]
I know this has been discussed before but I am still worried and
puzzled by the current semantics of get-childitem particularly when
used with the FileSystem provider.
One thing that worries me is the difference in semantics between
-Include and -Filter.
But why do we need both a -Filter and a -Include parameter at all? The
Help files are ... um ... unhelpful on that point as they are on the
exact semantics of either parameter. As is, so far as I can see, the
User Guide.

The Filter clause represents filtering in the native language of the
provider. In a SQL provider, you may have a filter clause like "LEFT OUTER
JOIN CustomerId ON ..."

Our wildcarding abilities are different that the native filters for all
providers.

Post by Andrew Watt [MVP]
Examples of why I'm unhappy follow below.
First, I did what I expected would be a simple comparison to be sure
that -Include and -Filter gave me the same results doing a
"straightforward" search for *.txt. But it didn't. I got a different
count as in the following two commands.
PS C:\PowerShellScripts> (get-childitem * -Include *.txt).count
27
PS C:\PowerShellScripts> (get-childitem * -Filter *.txt).count
28
[I found that the discrepant result was a file named odd.txtcls when I
had attempted a pipe and, presumably added a cls in error. But why do
-Incude and -Filter treat this, admittedl unusual case differently?]

This is because the "native filter language" of the Windows FileSystem is a
little more permissive. If you try 'dir *.txt' from cmd.exe, you'll see
that it returns the file that doesn't actually end in .txt.

For another example of the native filter language differing, see how it
matches against short file names!

[C:\temp\filtertest]
PS:94 > dir > "Long File Name.txt"

[C:\temp\filtertest]
PS:95 > dir > "Long File Name Also.txt"

[C:\temp\filtertest]
PS:96 > dir -Include *2*

[C:\temp\filtertest]
PS:97 > dir -Filter *2*


Directory: C:\temp\filtertest


Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 7/12/2006 11:01 AM 1056 Long File Name Also.txt

Post by Andrew Watt [MVP]
So far so puzzling but it becomes more puzzling. See the next four
commands.
PS C:\PowerShellScripts> (get-childitem -Include *.txt).count
PS C:\PowerShellScripts> (get-childitem -Filter *.txt).count
28
PS C:\PowerShellScripts> (get-childitem . -Include *.txt).count
PS C:\PowerShellScripts> (get-childitem . -Filter *.txt).count
28
Why does the -Filter count stay the same but the -Include count drops
to zero?

This is a definite usability problem that we are aware of. -Include
and -Exclude act upon the output of wildcard expansion, not on the output of
the command. "Get-ChildItem * -Include *.txt" first gets all the children
of the current directory, and then matches them against your wildcard of
'*'. On that wildcard expansion, we then Include the *.txt.

The four commands above do not use wildcard expansion. They request the
children of '.' -- the first command does it by default. Because of that,
we do not process the -Include and -Exclude arguments.

Post by Andrew Watt [MVP]
Another test was to see how -Filter behaved. In the following three
commands the same results were returned but in a different order when
* was used as the value for the (positional) Path parameter. Why
should the ordering of results be different?
PS C:\PowerShellScripts> get-childitem -Filter te*
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1
d---- 07/07/2006 20:35 Test1
d---- 18/05/2006 09:42 Test2
d---- 18/05/2006 09:42 Test3
PS C:\PowerShellScripts> get-childitem . -Filter te*
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1
d---- 07/07/2006 20:35 Test1
d---- 18/05/2006 09:42 Test2
d---- 18/05/2006 09:42 Test3
PS C:\PowerShellScripts> get-childitem * -Filter te*
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts
Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 07/07/2006 20:35 Test1
d---- 18/05/2006 09:42 Test2
d---- 18/05/2006 09:42 Test3
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1

These commands take different code paths depending on if we do wildcard
expansion or not. This has been fixed in our internal builds -- in one
path, we used to output directories and then files, and the other path, we
would output files then directories.

Post by Andrew Watt [MVP]
I then went on to compare the same three values for -Path (default, .
and *) using the -Include parameter and I got very different results
for the three commands.
Why do I only get two results for the following command? Why aren't
the folders test1, test2 and test3 displayed?
PS C:\PowerShellScripts> get-childitem * -Include te*
Microsoft.PowerShell.Core\FileSystem::C:\PowerShellScripts
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 20/05/2006 09:34 158 TenLines.txt
-a--- 11/05/2006 21:53 137 TenNumbers.ps1

Currently, -Include and -Exclude do not match against container names.

Post by Andrew Watt [MVP]
But, more confusing for me (at least) why do I get no results
displayed for the following two commands?
PS C:\PowerShellScripts> get-childitem -Include te*
PS C:\PowerShellScripts> get-childitem . -Include te*
PS C:\PowerShellScripts>

This is for the same reason above, in that -Include and -Exclude only act
against the result of globbing. These two commands don't perform globbing.

Post by Andrew Watt [MVP]
It seems to me that the semantics of the default path, . as a path and
* as a path are treated as the same when -Filter is used (except for
ordering of results in the second example) but the semantics of *
differs drastically from the default and . when -Include is used.
Now ... can somebody help to convince me that behaviour such as occurs
in the preceding examples makes good sense?

No :) Filter makes sense and applies globally -- I think the issue here was
a lack of clear documentation. -Include and -Exclude simply do not match
user's expectations. The Where-Object cmdlet is really your best bet to
perform consistent post-filtering of results. If you combine this with the
"Simple Where" function I wrote about the other day, it really becomes much
more intuitive:

[C:\temp\filtertest]
PS:133 > dir | wheres name like *t*


Directory: C:\temp\filtertest


Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 7/12/2006 11:16 AM Test
-a--- 7/12/2006 11:01 AM 1056 Long File Name Also.txt
-a--- 7/12/2006 11:01 AM 814 Long File Name.txt



[C:\temp\filtertest]
PS:134 > dir | wheres name notlike *t*

[C:\temp\filtertest]
PS:135 > dir | wheres name notlike *x*


Directory: C:\temp\filtertest


Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 7/12/2006 11:16 AM Test

Post by Andrew Watt [MVP]
Thanks
Puzzled of Tunbridge Wells :)
alias
Andrew Watt MVP

[Chris Warwick]

On Wed, 12 Jul 2006 11:26:34 -0700, "Lee Holmes [MSFT]"
<***@online.microsoft.com> wrote:


<..snip..>

Post by Lee Holmes [MSFT]
For another example of the native filter language differing, see how it
matches against short file names!
[C:\temp\filtertest]
PS:94 > dir > "Long File Name.txt"
[C:\temp\filtertest]
PS:95 > dir > "Long File Name Also.txt"
[C:\temp\filtertest]
PS:96 > dir -Include *2*
[C:\temp\filtertest]
PS:97 > dir -Filter *2*
Directory: C:\temp\filtertest
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 7/12/2006 11:01 AM 1056 Long File Name Also.txt

This has always been a HUGE problem IMHO, even with CMD.EXE. If, with
the above example files, you type "del *2*" it DELETES the file that
you thought was called "Long File Name Also.txt"!

(BTW, there's a setting in Group Policy to turn off the generation of
short file names - which you might think is a great idea considering
the above. Unfortunately this breaks lots of (admitedly badly
written) software, including SMS last time I checked - ho hum...)