DTrace and PHP, demonstrated

August 5, 2005

If you were at my presentation at OSCON yesterday, I apologize for the brisk pace — there was a ton of material to cover, and forty-five minutes isn’t much time. Now that I’ve got a little more time, I’d like to get into the details of the PHP demonstration that I did during the presentation. Here is the (very) simple PHP program that I was using:

<?php
function blastoff()
{
        echo "Blastoff!\\n\\n";
}

function one()
{
        echo "One...\\n";
        blastoff();
}

function two()
{
        echo "Two...\\n";
        one();
}

function three()
{
        echo "Three...\\n";
        two();
}

function launch()
{
        three();
}

while (1)
{
        launch();
        sleep(1);
}
?>

Running this in a window just repeats this output:

% php ./blastoff.php
php ./blastoff.php
Content-type: text/html
X-Powered-By: PHP/5.1.0-dev

Three...
Two...
One...
Blastoff!

Three...
Two...
One...
Blastoff!
...

Now, because I have specified Wez Furlong’s new dtrace.so as an extension in my php.ini file, when I run the above, two probes show up:

# dtrace -l | grep php
4   php806       dtrace.so          php_dtrace_execute function-return
5   php806       dtrace.so          php_dtrace_execute function-entry

The function-entry and function-return probes have three arguments:

arg0 is the name of the function (a pointer to a string within PHP)
arg1 is the name of the file containing the call site (also a pointer to a string within PHP)
arg2 is the line number of the call site a pointer to a string within PHP)

So for starters, let’s just get an idea of which functions are being called in my PHP program:

# dtrace -n function-entry'{printf("called %s() in %s at line %d\n", \
copyinstr(arg0), copyinstr(arg1), arg2)}' -q
called launch() in /export/php/blastoff.php at line 32
called three() in /export/php/blastoff.php at line 27
called two() in /export/php/blastoff.php at line 22
called one() in /export/php/blastoff.php at line 16
called blastoff() in /export/php/blastoff.php at line 10
called launch() in /export/php/blastoff.php at line 32
called three() in /export/php/blastoff.php at line 27
called two() in /export/php/blastoff.php at line 22
called one() in /export/php/blastoff.php at line 16
called blastoff() in /export/php/blastoff.php at line 10
^C

If you’re new to DTrace, note that you have the power to trace an arbitrary D expression in your action. For example, instead of printing out the file and line number of the call site, we could trace the walltimestamp:

# dtrace -n function-entry'{printf("called %s() at %Y\n", \
copyinstr(arg0), walltimestamp)}' -q
called launch() at 2005 Aug  5 08:08:24
called three() at 2005 Aug  5 08:08:24
called two() at 2005 Aug  5 08:08:24
called one() at 2005 Aug  5 08:08:24
called blastoff() at 2005 Aug  5 08:08:24
called launch() at 2005 Aug  5 08:08:25
called three() at 2005 Aug  5 08:08:25
called two() at 2005 Aug  5 08:08:25
called one() at 2005 Aug  5 08:08:25
called blastoff() at 2005 Aug  5 08:08:25
^C

Note, too, that (unless I direct it not to) this will aggregate across PHP instances. So, for example:

#!/usr/sbin/dtrace -s

#pragma D option quiet

php*:::function-entry
{
        @bypid[pid] = count();
        @byfunc[copyinstr(arg0)] = count();
        @bypidandfunc[pid, copyinstr(arg0)] = count();
}

END
{
        printf("By pid:\n\n");
        printa("  %-40d %@d\n", @bypid);

        printf("\nBy function:\n\n");
        printa("  %-40s %@d\n", @byfunc);

        printf("\nBy pid and function:\n\n");
        printa("  %-9d %-30s %@d\n", @bypidandfunc);
}

If I run three instances of blastoff.php and then the above script, I see the following after I ^C:

By pid:

  806                                      30
  875                                      30
  889                                      30

By function:

  launch                                   18
  three                                    18
  two                                      18
  one                                      18
  blastoff                                 18

By pid and function:

  875       two                            6
  875       three                          6
  875       launch                         6
  875       blastoff                       6
  889       blastoff                       6
  806       launch                         6
  889       one                            6
  806       three                          6
  889       two                            6
  806       two                            6
  889       three                          6
  806       one                            6
  889       launch                         6
  806       blastoff                       6
  875       one                            6

The point is that DTrace allows you to aggregate across PHP instances, allowing you to understand not just what a particular PHP is doing, but what PHP is doing more generally.

If we’re interested in better understanding the code flow in a particular PHP instance, we can write a script that uses a thread-local variable to follow the code flow:

#pragma D option quiet

self int indent;

php$target:::function-entry
/copyinstr(arg0) == "launch"/
{
        self->follow = 1;
}

php$target:::function-entry
/self->follow/
{
        printf("%*s", self->indent, "");
        printf("-> %s\n", copyinstr(arg0));
        self->indent += 2;
}

php$target:::function-return
/self->follow/
{
        self->indent -= 2;
        printf("%*s", self->indent, "");
        printf("follow = 0;
        exit(0);
}

Running the above requires giving the script a particular PHP process; assuming the above script is named blast.d, it might look like this:

# dtrace -s ./blast.d -p `pgrep -n php`
-> launch
  -> three
    -> two
      -> one
        -> blastoff
        <- blastoff
      <- one
    <- two
  <- three
<- launch

This shows us all of the PHP functions that were called from launch(), but it doesn’t tell the full story — we know that our PHP functions are calling into native code to do some of their work. To add this to the mix, we’ll write a slightly longer script:

#pragma D option quiet

self int indent;

php$target:::function-entry
/copyinstr(arg0) == "launch"/
{
        self->follow = 1;
}

php$target:::function-entry
/self->follow/
{
        printf("%\*s", self->indent, "");
        printf("-> %-20s %\*sphp\\n", copyinstr(arg0),
            46 - self->indent, "");
        self->indent += 2;
}

php$target:::function-return
/self->follow/
{
        self->indent -= 2;
        printf("%\*s", self->indent, "");
        printf("indent, "");
}

pid$target:libc.so.1::entry
/self->follow/
{
        printf("%\*s", self->indent, "");
        printf("-> %-20s %\*s%s\\n", probefunc, 46 - self->indent, "",
            probemod);
        self->indent += 2;
}

pid$target:libc.so.1::return
/self->follow/
{
        self->indent -= 2;
        printf("%\*s", self->indent, "");
        printf("indent, "",
            probemod);
}

php$target:::function-return
/copyinstr(arg0) == "launch"/
{
        self->follow = 0;
        exit(0);
}

Running the above yields the following output:

-> launch                                          php
  -> three                                         php
    -> write                                       libc.so.1
      -> _save_nv_regs                             libc.so.1
       _write                                    libc.so.1
      <- _write                                    libc.so.1
     two                                         php
      -> write                                     libc.so.1
        -> _save_nv_regs                           libc.so.1
         _write                                  libc.so.1
        <- _write                                  libc.so.1
       one                                       php
        -> write                                   libc.so.1
          -> _save_nv_regs                         libc.so.1
           _write                                libc.so.1
          <- _write                                libc.so.1
         blastoff                                php
          -> write                                 libc.so.1
            -> _save_nv_regs                       libc.so.1
             _write                              libc.so.1
            <- _write                              libc.so.1
          <- write                                 libc.so.1
        <- blastoff                                php
      <- one                                       php
    <- two                                         php
  <- three                                         php
<- launch                                          php

This shows us what’s really going on — or at least more of what’s really going on: our PHP functions are inducing work from libc. This is much more information, but of course, we’re still only seeing what’s going on at user-level. To add the kernel into the mix, add the following to the script:

fbt:genunix::entry
/self->follow/
{
        printf("%\*s", self->indent, "");
        printf("-> %-20s %\*skernel\\n", probefunc, 46 - self->indent, "");
        self->indent += 2;
}

fbt:genunix::return
/self->follow/
{
        self->indent -= 2;
        printf("%\*s", self->indent, "");
        printf("indent, "");
}

Running this new script generates much more output:

-> launch                                          php
  -> three                                         php
    -> write                                       libc.so.1
      -> _save_nv_regs                             libc.so.1
       _write                                    libc.so.1
        -> syscall_mstate                          kernel
          -> gethrtime_unscaled                    kernel
          <- gethrtime_unscaled                    kernel
         syscall_entry                           kernel
          -> pre_syscall                           kernel
           copyin_args32                         kernel
            -> copyin_nowatch                      kernel
              -> watch_disable_addr                kernel
              <- watch_disable_addr                kernel
            <- copyin_nowatch                      kernel
          <- copyin_args32                         kernel
         write32                                 kernel
          -> write                                 kernel
            -> getf                                kernel
              -> set_active_fd                     kernel
              <- set_active_fd                     kernel
             releasef                            kernel
              -> cv_broadcast                      kernel
              <- cv_broadcast                      kernel
            <- releasef                            kernel
          <- write                                 kernel
         syscall_mstate                          kernel
          -> gethrtime_unscaled                    kernel
          <- gethrtime_unscaled                    kernel
        <- syscall_mstate                          kernel
      <- _write                                    libc.so.1
     two                                         php
      -> write                                     libc.so.1
        -> _save_nv_regs                           libc.so.1
         _write                                  libc.so.1
          -> syscall_mstate                        kernel
            -> gethrtime_unscaled                  kernel
            <- gethrtime_unscaled                  kernel
          <- syscall_mstate                        kernel
          ...

Now we’re seeing everything that our PHP program is doing, from the PHP through the native code, into the kernel and back. So using DTrace on PHP has a number of unique advantages: you can look at the entire system, and you can look at the entire stack — and you can do it all in production. Thanks again to Wez for putting together the PHP provider — and if you’re a PHP developer, bon appetit!

22 Responses

Luke Closs says:

August 5, 2005 at 11:45 am

Great presentation Bryan. I had no opinion on OpenSolaris prior to this presentation, but had heard lots about dtrace. After your presentation, my coworkers and I were super stoked about opensolaris/dtrace. I can’t wait to install it, get our product (Sophos PureMessage) building on it and run dtrace on it!
As our project is largely perl based, we’d probably be interested in any effort to add probes to perl.
Great job! Your passion and excitement about dtrace were well evident from your presentation! It was one of my highlights from OSCON.
Luke

Log in to Reply
Matty says:

August 5, 2005 at 12:50 pm

Anyway Sun could add PHP to the Solaris 10 Apache2 package? Redhat ships with PHP, but for some reason Sun doesn’t. 🙁
– Ryan

Log in to Reply
Nik Clayton says:

August 5, 2005 at 5:14 pm

How big a deal is this?
The reason I ask is that after reading your post I started thinking about doing this sort of thing for Perl. Naturally, I started poking around the web to see if anyone else was already doing this.
That lead me to this interview at opensolaris.org. In that interview you say:

DTrace can’t currently instrument languages that have dynamic program text: Java, Python, Perl, PHP, etc. This is not an easy problem to solve: these languages have not been designed or implemented with dynamic instrumentation in mind, and the techniques required to instrument them are often very specific to a particular language and its run-time environment. Solving this problem still lies in the indefinite future for us–the next year or so will be spent evolving our existing functionality–but we very much intend to solve it.

and

Both. Instrumenting JITted code is difficult because the virtual machine needs to know what you’re up to at some level–even if only enough to know to leave your instrumentation alone. Once the code is instrumented, the granularity of operations makes it difficult to tie that instrumentation back to something meaningful. To give you a simple, concrete example: Perl 5 doesn’t associate line numbers with parse tree nodes unless it is explicitly started in an explicit debugging mode. By discarding that information, Perl makes it very difficult to [connect] Perl-induced system activity (I/O, CPU utilization, network activity, etc.) to the specific body of Perl that is inducing it. Programs written in C and C++ generally discard their debugging information too, but these languages are so much closer to the operating system and the underlying microprocessor that we can generally still draw meaningful inferences. Not so for Perl, Java, Python, PHP, etc.

That was less than two months ago.
I confess that I haven’t, yet, been able to use DTrace in the field. My exposure to it has been through information that I’ve read on the web, and sitting through a couple of Sun technical demonstrations about it in the UK. Those demos played up the ease with which it’s possible to write a DTrace provider, and walked through creating one. Without wishing to denigrate Wez’s work, I got the impression that simply instrumenting function entry points like this with a dynamic language like PHP or Perl is no great shakes.
Have a got the wrong end of the stick?

Log in to Reply
Alan Burlison says:

August 5, 2005 at 6:22 pm

Perl is possible, at least to the same level as the PHP stuff Wez has done, i.e. subroutine name, source line and file, see http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-08/msg00140.html.
Tracing individual opcode dispatch is also possible, if required.

Log in to Reply
Bryan Cantrill says:

August 5, 2005 at 6:43 pm

Nik, My comments a couple of months ago reflect the difficulty of truly dynamic instrumentation. That is, doing to VM-based languages what we can do with user-level C/C++. What we have discovered since is that imperfect solutions can be such a huge lurch forward for these environments that they are very much worth the time and trouble. The PHP solution, for example, doesn’t instrument every opcode. (And from talking to Marcus Boerger and Wez after my presentation, this is as hard as I claimed it to be.) But it’s such a huge leap, that it’s an effective solution — and it may even be “good enough” to obviate the need to solve the (much) harder problem of true dynamic instrumentation.

Log in to Reply
Alan Hargreaves' Weblog says:

August 6, 2005 at 11:35 pm

[Trackback] Following a link in a comment from Alan Burlinson from Bryan’s blog about creating a php provider , it appears that Alan is very well along the path to having a DTrace provider for perl.
To read a bit more about this, have a look at the mail…

Log in to Reply
Magnus Forsberg says:

October 11, 2005 at 1:03 pm

Hi Bryan,
The PHP Dtrace extension would really help me with a problem I got… I have a segfaulting PHP web application and can’t seem to narrow down the problem. So this extension would probably help me alot…
However I have some probs getting this thing to work:
# pear install dtrace
…
the only warning I can see is when linking:
*** Warning: Linking the shared library dtrace.la against the non-libtool
*** objects php.o is not portable!
… other than that the build seems to have gone through fine, even though it shouldn’t be more than 10k.
Build system:
Solaris Express build 20, on x86
Sun JDS CBE 1.2 tools, GCC 3.4.3
Apache 2.0.54
Latest PHP dev sources.
The output file gets huge:
204274 16416 -rwxr-xr-x 1 root root 16797924 Oct 11 21:53 /var/tmp/pear-build-defaultuser/install-dtrace-1.0.3/opt/php-STABLE-200510100832-gcc/lib/php/extensions/no-debug-non-zts-20041030/dtrace.so
when loading it from php.ini:
PHP Warning: PHP Startup: Unable to load dynamic library ‘/opt/php-STABLE-200510100832-gcc/lib/php/extensions/no-debug-non-zts-20041030/dtrace.so’ – ld.so.1: httpd: fatal: relocation error: file /opt/php-STABLE-200510100832-gcc/lib/php/extensions/no-debug-non-zts-20041030/dtrace.so: symbol zend_execute: referenced symbol not found in Unknown on line 0
nm output:
[59] | 0| 0|NOTY |GLOB |0 |UNDEF |zend_execute
[57] | 0| 0|NOTY |GLOB |0 |UNDEF |zend_execute_internal
[60] | 0| 0|NOTY |GLOB |0 |UNDEF |zend_get_executed_filename
[76] | 0| 0|NOTY |GLOB |0 |UNDEF |zend_get_executed_lineno
Any help would be greatly appreciated!

Log in to Reply
skjani basha says:

September 9, 2006 at 1:56 am

Respectes sir
Thank u for ur tutorials whihc u posted in net.It is very useful for me . Please send me sore worked programs with output. If u send these type or examples it is very useful for us.Please send me soon bezi depend on u r site and i am learning one by one PHP.
Thanks
your’s truely,
jani basha(india).

Log in to Reply
natural male enhancement says:

January 2, 2007 at 10:14 pm

natural male enhancement

Log in to Reply
xdtfsdtu says:

January 19, 2007 at 3:42 am

Offshore Outsourcing

Log in to Reply
wrwe4ty says:

January 19, 2007 at 3:47 am

Software Outsourcing

Log in to Reply
rytyty says:

January 19, 2007 at 3:49 am

Offshore Services

Log in to Reply
fhtrh says:

January 19, 2007 at 3:50 am

cheap flights sydney

Log in to Reply
gwergwerg says:

January 19, 2007 at 3:51 am

cheap flights johannesburg

Log in to Reply
gwergwerg says:

January 19, 2007 at 3:52 am

cheap flights johannesburg

Log in to Reply
qtyracks says:

April 4, 2007 at 1:43 am

[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=137]角钢货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=138]中型货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=145]中量型货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=153]重型货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=152]阁楼式货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=142]4S店货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=151]悬臂式货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=150]贯通式货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=154]抽屉式货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=168]压入式货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=167]移动式货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=170]线棒货架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=171]钢平台[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=166]密集架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=146]钢托盘[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=160]塑料托盘[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=176]木托盘[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=177]塑木托盘[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=147]置物架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=148]登高车[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=149]仓储笼[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=178]手推车[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=161]挂板架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=164]堆垛架[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=174]工作台[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=175]工具柜[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=173]周转箱[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=172]零件盒[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=163]物流台车[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=162]料箱[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=159]搬运机械[/url]
[url=http://www.quality-hj.com/Chinese/ProductShow.asp?ArticleID=158]堆高车[/url]
[url=http://www.quality-hj.com/Chinese/Product.asp]仓储货架[/url]
[url=http://www.quality-hj.com]货架[/url]
[url=http://www.quality-hj.com/Chinese/CoProfile.asp?Action=Contact]货架[/url]
[url=http://www.qtyracks.obm.cn/]货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=54838&Lid=54838&charset=0]货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56028&Lid=56028&charset=0]南京货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56029&Lid=56029&charset=0]仓储货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56030&Lid=56030&charset=0]货架公司[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56031&Lid=56031&charset=0]货架厂[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56032&Lid=56032&charset=0]宁波货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56033&Lid=56033&charset=0]北京货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56034&Lid=56034&charset=0]广州货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56035&Lid=56035&charset=0]仓库货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56036&Lid=56036&charset=0]仓储设备[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56037&Lid=56037&charset=0]重型货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56038&Lid=56038&charset=0]货架制造[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56040&Lid=56040&charset=0]托盘[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56041&Lid=56041&charset=0]仓储笼[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56042&Lid=56042&charset=0]物流设备[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56043&Lid=56043&charset=0]上海货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56044&Lid=56044&charset=0]货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56045&Lid=56045&charset=0]钢制托盘[/url]
[url=http://site.obm.cn/about.asp?site=qtyracks&id=56046&Lid=56046&charset=0]角钢货架[/url]
[url=http://site.obm.cn/index.asp?site=gdgs&charset=0]货架[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58714&Lid=58714&charset=0]货架[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58712&Lid=58712&charset=0]货架[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58797&Lid=58797&charset=0]轻型货架[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58798&Lid=58798&charset=0]重型货架[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58799&Lid=58799&charset=0]托盘[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58800&Lid=58800&charset=0]钢托盘[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58801&Lid=58801&charset=0]仓储笼[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58802&Lid=58802&charset=0]手推车[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58803&Lid=58803&charset=0]登高车[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58804&Lid=58804&charset=0]货架厂[/url]
[url=http://site.obm.cn/about.asp?site=gdgs&id=58805&Lid=58805&charset=0]仓储货架[/url]
[url=http://www.kfwh.com/blog.asp?name=admin]娱乐新闻[/url]
[url=http://blog.tyfo.com/?U=xwzx]娱乐新闻[/url]
[url=http://www.qtyhj.obm.cn/ ]货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60049&Lid=60049&charset=0]货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60051&Lid=60051&charset=0]仓储货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60052&Lid=60052&charset=0]货架厂[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60053&Lid=60053&charset=0]仓储设备[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60054&Lid=60054&charset=0]浙江货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60055&Lid=60055&charset=0]宁波货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60056&Lid=60056&charset=0]台州货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60057&Lid=60057&charset=0]温州货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60058&Lid=60058&charset=0]北京货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60059&Lid=60059&charset=0]上海货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60060&Lid=60060&charset=0]托盘[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60062&Lid=60062&charset=0]钢托盘[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60063&Lid=60063&charset=0]塑料托盘[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60064&Lid=60064&charset=0]轻型货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60065&Lid=60065&charset=0]重型货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60066&Lid=60066&charset=0]角钢货架[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60067&Lid=60067&charset=0]手推车[/url]
[url=http://site.obm.cn/about.asp?site=qtyhj&id=60068&Lid=60068&charset=0]杭州货架[/url]
[url=http://blog.hc360.com/portal/personArticleListSplit.do?blogName=qtyrack]货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116999]货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116997]托盘[/url]
[url=http://hblog.hc360.com/portal/personShowArticle.do?articleId=116995]仓储笼[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116994]青岛货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116993]济南货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116992]沈阳货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116991]天津货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116990]上海货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116988]北京货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116987]台州货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116985]温州货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116984]宁波货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116983]杭州货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116981]浙江货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116977]仓储笼[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116975]钢托盘[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116973]物流设备[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116969]塑料托盘[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116968]南京货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116967]货架公司[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116966]货架厂[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116965]仓库货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116963]仓储设备[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116962]仓储货架[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116959]手推车[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116958]托盘[/url]
[url=http://blog.hc360.com/portal/personShowArticle.do?articleId=116953]货架[/url]

Log in to Reply
qtyracks says:

April 4, 2007 at 1:46 am

[http://www.qtyracks.obm.cn/ 货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=54838&Lid=54838&charset=0 货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56028&Lid=56028&charset=0 南京货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56029&Lid=56029&charset=0 仓储货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56030&Lid=56030&charset=0 货架公司]
[http://site.obm.cn/about.asp?site=qtyracks&id=56031&Lid=56031&charset=0 货架厂 ]
[http://site.obm.cn/about.asp?site=qtyracks&id=56032&Lid=56032&charset=0 宁波货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56033&Lid=56033&charset=0 北京货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56034&Lid=56034&charset=0 广州货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56035&Lid=56035&charset=0 仓库货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56036&Lid=56036&charset=0 仓储设备]
[http://site.obm.cn/about.asp?site=qtyracks&id=56037&Lid=56037&charset=0 重型货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56038&Lid=56038&charset=0 货架制造]
[http://site.obm.cn/about.asp?site=qtyracks&id=56040&Lid=56040&charset=0 托盘]
[http://site.obm.cn/about.asp?site=qtyracks&id=56041&Lid=56041&charset=0 仓储笼]
[http://site.obm.cn/about.asp?site=qtyracks&id=56042&Lid=56042&charset=0 物流设备]
[http://site.obm.cn/about.asp?site=qtyracks&id=56043&Lid=56043&charset=0 上海货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56044&Lid=56044&charset=0 货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56045&Lid=56045&charset=0 钢制托盘]
[http://site.obm.cn/about.asp?site=qtyracks&id=56046&Lid=56046&charset=0 角钢货架]
[http://site.obm.cn/index.asp?site=gdgs&charset=0 货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58714&Lid=58714&charset=0 货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58712&Lid=58712&charset=0 货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58797&Lid=58797&charset=0 轻型货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58798&Lid=58798&charset=0 重型货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58799&Lid=58799&charset=0 托盘]
[http://site.obm.cn/about.asp?site=gdgs&id=58800&Lid=58800&charset=0 钢托盘]
[http://site.obm.cn/about.asp?site=gdgs&id=58801&Lid=58801&charset=0 仓储笼]
[http://site.obm.cn/about.asp?site=gdgs&id=58802&Lid=58802&charset=0 手推车]
[http://site.obm.cn/about.asp?site=gdgs&id=58803&Lid=58803&charset=0 登高车]
[http://site.obm.cn/about.asp?site=gdgs&id=58804&Lid=58804&charset=0 货架厂]
[http://site.obm.cn/about.asp?site=gdgs&id=58805&Lid=58805&charset=0 仓储货架]
[http://www.kfwh.com/blog.asp?name=admin 娱乐新闻]
[http://www.qtyhj.obm.cn/ 货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60049&Lid=60049&charset=0 货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60051&Lid=60051&charset=0 仓储货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60052&Lid=60052&charset=0 货架厂]
[http://site.obm.cn/about.asp?site=qtyhj&id=60053&Lid=60053&charset=0 仓储设备]
[http://site.obm.cn/about.asp?site=qtyhj&id=60054&Lid=60054&charset=0 浙江货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60055&Lid=60055&charset=0 宁波货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60056&Lid=60056&charset=0 台州货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60057&Lid=60057&charset=0 温州货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60058&Lid=60058&charset=0 北京货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60059&Lid=60059&charset=0 上海货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60060&Lid=60060&charset=0 托盘]
[http://site.obm.cn/about.asp?site=qtyhj&id=60062&Lid=60062&charset=0 钢托盘]
[http://site.obm.cn/about.asp?site=qtyhj&id=60063&Lid=60063&charset=0 塑料托盘]
[http://site.obm.cn/about.asp?site=qtyhj&id=60064&Lid=60064&charset=0 轻型货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60065&Lid=60065&charset=0 重型货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60066&Lid=60066&charset=0 角钢货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60067&Lid=60067&charset=0 手推车]
[http://site.obm.cn/about.asp?site=qtyhj&id=60068&Lid=60068&charset=0 杭州货架]
[http://blog.hc360.com/portal/personArticleListSplit.do?blogName=qtyrack 货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116999 货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116997 托盘]
[http://hblog.hc360.com/portal/personShowArticle.do?articleId=116995 仓储笼]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116994 青岛货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116993 济南货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116992 沈阳货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116991 天津货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116990 上海货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116988 北京货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116987 台州货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116985 温州货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116984 宁波货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116983 杭州货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116981 浙江货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116977 仓储笼]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116975 钢托盘]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116973 物流设备]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116969 塑料托盘]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116968 南京货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116967 货架公司]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116966 货架厂]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116965 仓库货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116963 仓储设备]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116962 仓储货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116959 手推车]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116958 托盘]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116953 货架]

Log in to Reply
qtyracks says:

April 4, 2007 at 1:52 am

[http://www.qtyracks.obm.cn/ 货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=54838&Lid=54838&charset=0 货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56028&Lid=56028&charset=0 南京货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56029&Lid=56029&charset=0 仓储货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56030&Lid=56030&charset=0 货架公司]
[http://site.obm.cn/about.asp?site=qtyracks&id=56031&Lid=56031&charset=0 货架厂 ]
[http://site.obm.cn/about.asp?site=qtyracks&id=56032&Lid=56032&charset=0 宁波货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56033&Lid=56033&charset=0 北京货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56034&Lid=56034&charset=0 广州货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56035&Lid=56035&charset=0 仓库货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56036&Lid=56036&charset=0 仓储设备]
[http://site.obm.cn/about.asp?site=qtyracks&id=56037&Lid=56037&charset=0 重型货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56038&Lid=56038&charset=0 货架制造]
[http://site.obm.cn/about.asp?site=qtyracks&id=56040&Lid=56040&charset=0 托盘]
[http://site.obm.cn/about.asp?site=qtyracks&id=56041&Lid=56041&charset=0 仓储笼]
[http://site.obm.cn/about.asp?site=qtyracks&id=56042&Lid=56042&charset=0 物流设备]
[http://site.obm.cn/about.asp?site=qtyracks&id=56043&Lid=56043&charset=0 上海货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56044&Lid=56044&charset=0 货架]
[http://site.obm.cn/about.asp?site=qtyracks&id=56045&Lid=56045&charset=0 钢制托盘]
[http://site.obm.cn/about.asp?site=qtyracks&id=56046&Lid=56046&charset=0 角钢货架]
[http://site.obm.cn/index.asp?site=gdgs&charset=0 货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58714&Lid=58714&charset=0 货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58712&Lid=58712&charset=0 货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58797&Lid=58797&charset=0 轻型货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58798&Lid=58798&charset=0 重型货架]
[http://site.obm.cn/about.asp?site=gdgs&id=58799&Lid=58799&charset=0 托盘]
[http://site.obm.cn/about.asp?site=gdgs&id=58800&Lid=58800&charset=0 钢托盘]
[http://site.obm.cn/about.asp?site=gdgs&id=58801&Lid=58801&charset=0 仓储笼]
[http://site.obm.cn/about.asp?site=gdgs&id=58802&Lid=58802&charset=0 手推车]
[http://site.obm.cn/about.asp?site=gdgs&id=58803&Lid=58803&charset=0 登高车]
[http://site.obm.cn/about.asp?site=gdgs&id=58804&Lid=58804&charset=0 货架厂]
[http://site.obm.cn/about.asp?site=gdgs&id=58805&Lid=58805&charset=0 仓储货架]
[http://www.kfwh.com/blog.asp?name=admin 娱乐新闻]
[http://www.qtyhj.obm.cn/ 货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60049&Lid=60049&charset=0 货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60051&Lid=60051&charset=0 仓储货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60052&Lid=60052&charset=0 货架厂]
[http://site.obm.cn/about.asp?site=qtyhj&id=60053&Lid=60053&charset=0 仓储设备]
[http://site.obm.cn/about.asp?site=qtyhj&id=60054&Lid=60054&charset=0 浙江货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60055&Lid=60055&charset=0 宁波货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60056&Lid=60056&charset=0 台州货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60057&Lid=60057&charset=0 温州货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60058&Lid=60058&charset=0 北京货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60059&Lid=60059&charset=0 上海货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60060&Lid=60060&charset=0 托盘]
[http://site.obm.cn/about.asp?site=qtyhj&id=60062&Lid=60062&charset=0 钢托盘]
[http://site.obm.cn/about.asp?site=qtyhj&id=60063&Lid=60063&charset=0 塑料托盘]
[http://site.obm.cn/about.asp?site=qtyhj&id=60064&Lid=60064&charset=0 轻型货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60065&Lid=60065&charset=0 重型货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60066&Lid=60066&charset=0 角钢货架]
[http://site.obm.cn/about.asp?site=qtyhj&id=60067&Lid=60067&charset=0 手推车]
[http://site.obm.cn/about.asp?site=qtyhj&id=60068&Lid=60068&charset=0 杭州货架]
[http://blog.hc360.com/portal/personArticleListSplit.do?blogName=qtyrack 货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116999 货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116997 托盘]
[http://hblog.hc360.com/portal/personShowArticle.do?articleId=116995 仓储笼]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116994 青岛货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116993 济南货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116992 沈阳货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116991 天津货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116990 上海货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116988 北京货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116987 台州货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116985 温州货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116984 宁波货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116983 杭州货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116981 浙江货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116977 仓储笼]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116975 钢托盘]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116973 物流设备]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116969 塑料托盘]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116968 南京货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116967 货架公司]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116966 货架厂]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116965 仓库货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116963 仓储设备]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116962 仓储货架]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116959 手推车]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116958 托盘]
[http://blog.hc360.com/portal/personShowArticle.do?articleId=116953 货架]

Log in to Reply
二手车 says:

May 7, 2007 at 12:24 am

租车|快来租提供(上海租车二手车|)服务,专业汽车租赁公司. Google排名翻译公司北京翻译公司北京翻译公司上海翻译公司  深圳翻译公司  广州翻译公司北京印刷公司印刷小游戏 flash小游戏休闲小游戏美眉小游戏化妆小游戏休闲小游戏 mm小游戏迷你小游戏上海办公用品北京办公用品办公用品北京整形医院机票数据恢复成人用品减肥注册公司注册香港公司注册香港公司租车数据恢复搬家整形数据恢复小游戏下载休闲小游戏  迷你小游戏标签  在线小游戏网站优化搬场机票轴承阀门室内设计深圳租车留学

Log in to Reply
二手车 says:

May 7, 2007 at 12:26 am

窃听器手机窃听器手机窃听器监听器监听器手机监听器手机监听器手机监听器手机窃听器窃听器窃听器  窃听器  手机窃听器  手机监听器  监听器  监听器  手机监听器  手机窃听器  窃听器  手机窃听器监听器手机监听器窃听器  监听器手机窃听器手机窃听器手机监听器监听器窃听器  窃听器手机窃听器手机监听器手机监听器窃听器手机窃听器手机窃听器手机窃听器手机窃听器手机窃听器手机窃听器
 窃听器手机窃听器监听器手机监听器

Log in to Reply
迷途的小羔羊 says:

June 26, 2007 at 5:47 pm

[URL=http://hi.baidu.com/sent2008/blog/item/40fc05fb328eb0176c22eb6d.html]货架[/URL]
[URL=http://hi.baidu.com/sent2008/blog/item/72225f2470d02a32c995596e.html]角钢货架[/URL]
[URL=http://hi.baidu.com/sent2008/blog/item/1eb632d0807b628ea1ec9c68.html]轻型货架[/URL]
[URL=http://hi.baidu.com/sent2008/blog/item/3e38d2268586ed168b82a16a.html]中型货架[/URL]
[URL=http://hi.baidu.com/sent2008/blog/item/da7e9e1b456c0efbae51336a.html]重型货架[/URL]
[URL=http://hi.baidu.com/sent2008/blog/item/67dd010f4185242b6159f374.html]悬臂货架[/URL]
[URL=http://hi.baidu.com/sent2008/blog/item/b26d4e17c6ff4a004b90a775.html]模具货架[/URL]

Log in to Reply
迷途的小羔羊 says:

June 26, 2007 at 5:56 pm

[URL=http://www.ent-u.com/supply/detail/1132764.html]移动工具车[/URL]
[URL=http://www.ent-u.com/supply/detail/1132764.html]工具车/a>
[URL=http://tw.netsh.com/eden/blog/ctl_eden_blog.php?ctlAct=get&ctlObj=blog_log&iLogID=330005]油桶车[/URL]
[URL=http://tw.netsh.com/eden/blog/ctl_eden_blog.php?ctlAct=get&ctlObj=blog_log&iLogID=330004]全电动搬运车[/URL]

Log in to Reply

DTrace and PHP, demonstrated

22 Responses

Leave a Reply Cancel reply

Recent Posts

Archives

Archives