All SPEC CPU results for AMD Barcelona invalidated

All SPEC CPU results for AMD Barcelona invalidated

Summary: As of today, all of AMD's Barcelona SPEC CPU results so far have just been invalidated and labeled "Not-Available" by SPEC due to violation of the 3-month availability clause.  AMD is struggling with availability of its new quad-core “Barcelona” chips because of yield problems in manufacturing and the newly disclosed TLB (Translation Lookaside Buffer) bug.

SHARE:

As of today, all of AMD's Barcelona SPEC CPU results so far have just been invalidated and labeled "Not-Available" by SPEC due to violation of the 3-month availability clause.  AMD is struggling with availability of its new quad-core “Barcelona” chips because of yield problems in manufacturing and the newly disclosed TLB (Translation Lookaside Buffer) bug.  The TLB bug which causes a performance penalty when patched has forced AMD to stop shipping the Opteron chips to all but a few customers and it is impacting AMD financials.

The TLB bug also affects all AMD "Phenom" branded desktop-variant Barcelona processors though AMD is continuing to launch that part because of the rare nature of the problem.  Customers will have the option to download the BIOS patch at a later time from their motherboard manufacturer.  AMD has confirmed that the fix will come with the next revision called stepping B3 due out in Q1-2008.

Topics: CXO, Hardware, Processors, Software, IT Employment

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

40 comments
Log in or register to join the discussion
  • Invalidated on a technicality

    not on any actual problem with the results per se.

    Not that you'd ever take an opportunity to sling mud at AMD mind you...
    Robert Crocker
    • Not just a technicality

      You do realize that there is a 13+ percent performance hit on average desktop applications when the TLB patch is implemented right? Furthermore, there is a 50% performance hit on the memory bandwidth with the TLB patch and that would severely alter the SPECfp_rate2006 results because it's so heavily weighted on memory bandwidth. So you can't honestly tell me the current results are valid for pre-B3 stepping parts can you?

      They will become valid when B3 launches and that's a ways off given the difficulty in yielding higher clocked parts like the 2.5 2360SE. So at this point in time these results do not represent anything you can buy and SPEC CPU is about what you can buy within 3 months of the submission. Otherwise what is to prevent companies from submitting results for product that won't launch for 6 months or even a year? What SPEC has done is absolutely fair.
      georgeou
      • Did the spec program finish?

        Obviously they ran the SPEC code on the CPU and got results.

        Now, they've since discovered an errata in the chip that will occasionally cause it to hang up so they're going back to try and fix it. If the chip successfully finished the suite and produced the results then they are valid for that.

        Does this mean that you'd want to run the chip in that mode? I guess that's one of those cost/benefit decisions that the person using the chip would have to make.
        Robert Crocker
        • Guess again.

          "I guess that's one of those cost/benefit decisions that the person using the chip would have to make."

          The chip fails to perform its basic function under [u]normal[/u] operating conditions. It's garbage. The allowances you're trying to make would be reasonable, for the first microchip ever invented, assuming a likely approach to solving them, but this is not the first microchip ever invented. This is a chip that should have been tested better before release, plain and simple.
          Absolutely
        • Read the SPEC Docuementation

          http://www.spec.org/cpu2006/Docs/runrules.html

          Section 4.1
          " a tester publishes results for a hardware or software configuration that has not yet shipped,

          The component suppliers must have firm plans to make production versions of all components generally available, within 3 months of the first public release of the result (whether first published by the tester or by SPEC); and

          The tester must specify the general availability dates that are planned.

          Note 1: "Generally available" is defined in the SPEC Open Systems Group Policy document, which can be found at http://www.spec.org/osg/policy.html.

          Note 2: It is acceptable to test larger configurations than customers are currently ordering, provided that the larger configurations can be ordered and the company is prepared to ship them."

          IF they are found in violation of the 90 day availability the remedy is this:
          http://www.spec.org/osg/policy.html

          "The subcommittee may also [vote to] undertake a re-review of a previously published result, if significant questions as to its adherence to the applicable run and reporting rules arise. The OSSC reserves the ability to denote a publish result as non-compliant if it has been found in violation of the run and reporting rules.

          The following process has been established to handle instances where run and reporting rule violations are discovered in results that have been published on SPEC's web site.

          The subcommittee is responsible for identifying that a result published on the public web site is in violation of their benchmark run and reporting rules. Once a subcommittee identifies such a result, the result may re-enter a review process unless the submitter agrees with the finding of non-compliance.

          If the subcommittee's review returns a finding that the result is not in compliance with the benchmark's run and reporting rules, the following should occur.
          The results are modified:

          To remove the metric values (summary & individual) and replace with "NC" (non-compliant) .
          Add a header to the disclosure stating that the result did not comply to the rules and the reason(s) why.
          The submitter, with the subcommittee's approval of the text, may add additional wording into a remedy section. The remedy section is expected to explain how the vendor has addressed the problem. For example, mentioning that a new result has been submitted on the same platform with the issue resolved is the expected typical usage of this section.
          2.3.3 Handling Continued Availability in Published Results
          The various SPEC benchmarks' Run and Reporting Rules have clauses that at a certain time after the first publication (on SPEC's web site or otherwise), all components of the System Under Test (SUT) must be available for general customer shipment. The purpose of this requirement is

          to ensure that the results refer to a real, available system;
          to enable others to verify reproduce-ability of the results.
          For the case that this availability status cannot be maintained over a minimum time length, the following rules apply:

          Either at the request of the test sponsor or as a result of a subcommittee resolution, the subcommittee can undertake a re-review of a previously published result if a performance-relevant component ceases to be available for 30 days or more within 90 days after initial overall availability (i.e. initial availability of all components of the SUT). If the subcommittee's review returns a finding that this non-availability condition holds for a performance-relevant component, the result is marked "Not Available" (NA) in the following way:

          The metric values (summary and individual) are removed and replaced with "NA" (not available).
          A header is added to the disclosure stating that the result is for a system that is, at a time specified in the header, not available.
          The test sponsor can, with the subcommittee's approval of the text, add additional wording into a remedy section. The remedy section can for example, identify the part that is not available, contain the reason for non-availability, and/or an estimate when all components of the SUT are expected to be available again. If and when the SUT becomes available, the submitter may notify the subcommittee that the product is available. With the Subcommittee's agreement, the NA marking will be removed from the result page, however the page will retain the notice that the page was marked NA along with the history of the NA marking. This restarts the clock for the continuous availability requirement covered in this section.
          For results not published by SPEC, the subcommittee can ask the publisher to withdraw the publication.

          "Performance-relevant" is defined in the various benchmarks' Run and Reporting Rules' reproduceability clauses.

          Replacement by a new similar component with equivalent or better performance is considered acceptable and not a reason for an NA marking.

          In cases where the Run Rules allow use of a beta product, this beta product or a later equivalent or better performing regular product (see clause above) must remain available.

          Note: Effective for results published on or after July 1, 2002.

          "

          So if they publish a score and never ship, it is non-compliant (this was IBM's case), if they had some product on the market for a short time but withdrew, they are "Not Available", SPEC does not remove the entry but keeps it in the database, in the public domain. There is a reason for this... to dissuage others from submitting and publishing benchmark data that has no merit on systems that cannot be bought. This is the 'penalty' box for bad submissions.
          JumpingJack
      • Fair?

        Why didn't Spec just state the test results are suspended instead of invalidating? That would have caused less problems for AMD. No, sorry, I'm not an AMD fan, but common sense shows that Spec overstepped their bounds on this one and has unfairly damaged AMD.
        Narg
        • Again... read Spec's policies and guidelines...

          It is not that simple, what spec did was simply following the rules and policies they outlined as part of thier charter .. all the major industry players who submit scores purchase and agree to the licensing agreement, spec is very clear what they do to scores submitted and published that do not adhere to their rules for submissions.

          http://www.spec.org/osg/policy.html
          See section 2.3.2 Handling OSG Run Rule Violations in Published Results for what they do when a violatin occurs.

          See section 2.3.3 Handling OSG Run Rule Violations in Published Results for what they do when a vendor violated the continued availability.

          Anyone submitting scores has agreed to these rules, if AMD were not to have botched this launch and product so badly then spec would not need to invoke the rule.

          I can see how people are taken a back by how SPEC labeled the results as such though, however, searching their DB, this is the first time they had to invoke this rule for SPEC2006 CPU scores.

          One could say AMD is innovating and blazing a new path with another first :) ...
          JumpingJack
    • Invalidated based on unavailability

      SPEC (not George) invalidated the benchmarks because the chips are not available. No technicality involved.

      Basically SPEC (and I think this is quite reasonable) says you can't use SPEC benchmarks on a chip that isn't really in mass-production. There is no sense in comparing the performance of chip you can't get.
      DevGuy_z
  • I have ONE word for SPEC's approach to handling this

    .
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS
    OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS OBNOXIOUS



    P.S. It's really a shame and a great disappointment to see AMD screw up over and over and over. If they don't get their act together soon, even I will be buying Intel CPUs because AMD will not be meeting my absolute performance requirements at any price.
    TechExec2
    • That was 48 words

      That was 48 words, 47 over. I don't think you can blame this on SPEC, I've never seen this happen before on SPEC (though it may have happened before). SPEC has rules and vendors are expected to follow them.
      georgeou
      • I didn't blame this on SPEC

        I didn't blame this on SPEC. I said SPEC handled this in an obnoxious way. SPECs overprinting of "Unavailable" like that was obnoxious and un-called for. If AMD violated SPECs rules in a material way, AMD should get the smack down. But, not like that.
        TechExec2
        • Spec's Procedure

          When SPEC determines that a company or vendor is in non-compliance, the are clear on what they do:

          http://www.spec.org/osg/policy.html

          <i>If the subcommittee's review returns a finding that the result is not in compliance with the benchmark's run and reporting rules, the following should occur.

          The results are modified:

          1. To remove the metric values (summary & individual) and replace with "NC" (non-compliant) .

          2. Add a header to the disclosure stating that the result did not comply to the rules and the reason(s) why.

          3. The submitter, with the subcommittee's approval of the text, may add additional wording into a remedy section. The remedy section is expected to explain how the vendor has addressed the problem. For example, mentioning that a new result has been submitted on the same platform with the issue resolved is the expected typical usage of this section.
          </i>

          This is the smack down .... the 'penalty box' for submitting scores that do not adhere to the running and reporting rules.
          JumpingJack
    • It's not obnoxious

      because AMD doesn't have a shipping solution that provides an equivalent level of performance. They could run the benchmarks again with the TLB workaround, but because of the performance hit that wouldn't be very favorable for them either.
      t_mohajir
      • Look up "obnoxious"

        .
        Look up "obnoxious". It doesn't mean "invalid". Assuming AMD materially violated SPEC rules, SPEC should invalidate those AMD test results.
        TechExec2
        • How material do you want to get?

          They broke one of the most important rules of SPEC, which is a requirement to not be vaporware. "Not Available" is essentially the same as labeling it as "Vaporware".
          georgeou
          • Not vaporware

            Not by ANY means is the chip vaporware.

            It exists and it runs but it has errata. AMD is doing the right thing and fixing it.
            Robert Crocker
          • re: Not vaporware

            [i]It exists and it runs but it has errata. AMD is doing the right thing and fixing it.[/i]

            Agreed, but SPEC is doing the right thing as well. I'm sure AMD can re-test and certify when the problems are resolved.
            Badgered
          • The AMD SPEC test results are not invalid

            The AMD SPEC test results are not invalid. The AMD CPUs perform as tested. The problem is only that AMD has violated one of SPECs little rules about product availability.

            I don't see SPEC's smack-down of AMD as personal or biased. I expect SPEC would smack-down Intel in the same way if Intel broke one of SPEC's rules. But, the WAY they are doing it is obnoxious. Break their little rules and they will obnoxiously remind the entire world that THEY run the SPEC benchmark. And, this clearly delights George Ou who just LOVES obnoxious AMD smack-downs.

            AMD is screwing up ... again. There is no news here. Move along. Move along.
            TechExec2
          • Intel knows the SPEC-approved way to "ship" a vaporware processor

            .
            It appears that Intel knows the SPEC-approved way to "ship" a vaporware processor. AMD should take a lesson here.

            In April 2007, Intel announced the "availability" of their 3.0 GHz dual-dual core "quad" processor. And, Intel got lots of people to talk about the CPU as if it were actually available when it really was not. The part could not be found for purchase anywhere for months. But Intel did not post any SPEC benchmark tests until June. The CPU finally did appear for sale on the Internet in August.
            TechExec2
          • You are seriously not in "Tech"nology..

            ..my co-workers are laughing at you.

            Please stop - we are busting out in laughter and can't get back to work.

            You ought to sign-up for open mike at the Comedy Store some day.

            Here is an example of what you wrote earlier that makes us all laugh

            "I expect SPEC would smack-down Intel in the same way if Intel broke one of SPEC's rules. But, the WAY they are doing it is obnoxious. [b]Break their little rules and they will obnoxiously remind the entire world that THEY run the SPEC benchmark.[/b]"

            It just so happens that [b]their little rules[/b] were agreed to by the SPEC members, including AMD.

            LOL at you. Better luck next time, Mr Yager.
            thetruthhurts