PLDI and PACM

Introduction

Proceedings of the ACM on Programming Languages (PACMPL) is a new open access journal that will publish the proceedings of POPL, ICFP, and OOPSLA. Should PLDI join PACMPL? The purpose of this blog is to explain more about PACMPL, to present the cases for and against PLDI joining the journal, and to provide an online forum for discussion of the decision. There will also be an open discussion at the Town Hall Meeting at PLDI’17 in Barcelona (5pm on Tuesday 20th June). The PLDI Steering Committee is running a survey to take the pulse of the Community on this issue; if you have an interest, please do complete the survey! Based on the information and opinions gathered in the survey, on this blog, and at the Town Hall Meeting, PLDI SC will decide whether PLDI should join PACMPL.

About PACMPL

PACMPL was established to publish the proceedings of the top SIGPLAN conferences. The only significant change from the usual conference reviewing process is the introduction of a light second phase, whereby the PC determines any mandatory revisions among the usual suggestions for improvement, and formally checks that these have been carried out; PLDI already has a shepherding process that is essentially equivalent. PACMPL papers will be published with Gold Open Access (OA) from the outset. ICFP’17 will be the first issue of PACMPL; from that point, ICFP, OOPSLA, and POPL will all be published in the journal. There is the opportunity to include PLDI too.

Why PLDI should join PACMPL

  • For researchers in some jurisdictions—especially in Eastern Europe and South America—conference publications carry little weight in assessment and promotion exercises, even when top conferences are compared to second-rate journals. PLDI publications are at least as strong and as thoroughly reviewed as good journal publications. Calling them journal publications would allow researchers in this situation to align their scientific incentives (to publish in the best places) with their professional ones (to publish in indexed journals).
  • The PLDI Community has already expressed its support in principle for publishing the proceedings of PLDI in a journal. Two surveys in 2014 showed strong support for publishing the proceedings as an issue of TOPLAS: a general survey sent to PLDI’14 attendees was 75% in favour (N=91), and a subsequent targeted survey 70% in favour (N=212). (PLDI SC and SIGPLAN Executive Committee put a proposal to this effect to ACM Publications Board, but it was turned down. It was partly that rejection that led eventually to the new PACM journal series, and to PACMPL in particular.)
  • A survey sent to PLDI'15 attendees asked about inclusion in PACM in particular. 234 people answered, with 63% for, 27% against, and 9% unsure or indifferent. Breaking down those in favour by the number of authored PLDI papers results thus: 0 PLDI papers: 81% in favour; 1 PLDI paper: 59% in favour; 2 PLDI papers: 56%; 3 PLDI papers: 51%; 4 PLDI papers: 46%; >= 5 PLDI papers: 44%. The results show that the more PLDI publications you have, the less you support publishing PLDI in PACMPL; the last two groups comprise about 20% of those surveyed. A key question is how to interpret these results. One reading is that the older people in top universities are fine with the status quo, because the problem doesn't affect them, while the younger ones and everyone else are feeling the pain. Yet another view is geographic. Senior PLDI authors are heavily US-based. Most (though not all) of the pain reports of the conference-centered publication model come from the rest of the world. 
  • SIGPLAN members are strongly in favour of Open Access publication. A January 2017 SIGPLAN poll showed broad support for doing so (65% in favour, N=305). Gold OA in PACMPL will be provided at a reduced fee of $400 per paper for the whole conference (as opposed to $700 per paper individually), and SIGPLAN have agreed to underwrite this cost for the first three years of PACMPL. 
  • Citations to papers will be of the form “PACMPL, Volume 1, Number PLDI”, thus preserving the individual conference identity—PLDI papers will still be recognizably such in bibliographies and CVs.
  • If PLDI were to join PACMPL from 2018, then all four of the top SIGPLAN conferences would be included from the same point in time (namely, everything after July 2017).
  • If PLDI waits to join, PLDI authors and readers will miss out on Gold OA in the meantime. 
  • Publication in PACMPL of a PLDI paper reporting the latest results does not preclude the current option for later publication of a longer paper in a journal such as TOPLAS expanding on those results, or integrating the results from a series of conference papers.

Why PLDI should not join PACMPL

  • The premise for this proposal is that the quality of a researcher’s output be judged according to the venue in which the author’s work appears.   The fallacy of this approach to research assessment has been thoroughly and systematically addressed by prestigious journals and numerous international research organizations, including DORA, the UK Parliament, the European Association of Science Editors, the American Society for Biology and others.  Recommendation 1 of the DORA statement reads:  “Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.”  The proposal to move to PACMPL is thus based on a deeply flawed assessment model. Furthermore, the fallacy of this premise is gaining wide acceptance, and richer, easily accessible measures of research impact are now widely available.
  • It is not certain that commercial journal indexing services such as Clarivate (the new owners of what was formerly Thomson Web of Science) will actually index PACMPL as a journal.  If PACMPL is not indexed by Clarivate, then it will not appear in the Journal Citation Reports, which is regarded as the authoritative listing of journal impact factors.
  • If PACMPL is recognized as a journal, its capacity to achieve a meaningful impact factor is limited by the number of citations to PACMPL articles from recognized sources, which includes SIGPLAN Notices but excludes conference publications including LNCS (ISSN  0302-9743).
  • Consistent with the above, SIGPLAN has for a long time maintained that the binary distinction between journal and conference is orthogonal to the quality and impact of published work. Having won that argument, it would be perverse to abandon it, particularly at a time when others are publicly moving in the opposite direction. If PLDI were to join PACMPL, it would send a clear message that high quality publications appear in journals, while conferences are for second-tier work.   This will diminish the perceived value of SIGPLAN’s conferences, many of which routinely publish high impact work, harming a core element of SIGPLAN’s mission. On the other hand, as long as PLDI continues to publish as a conference (alongside ASPLOS and other high impact venues), it sends a strong message that both journals and conferences are valid modes of publication for high impact work.
  • Support for publication in TOPLAS does not imply support for publication in PACMPL. For one thing, TOPLAS is an established journal with a strong reputation, whereas PACMPL is new. For a second, there was no proposal that any other conference would have shared TOPLAS with PLDI; whereas PACMPL will be shared with POPL, ICFP, and OOPSLA from the outset, with the possibility of further SIGPLAN conferences such as PPoPP joining later.
  • An alternative interpretation of the PLDI’15 survey results is that the senior members of the community have more experience about what works, and are hesitant to change a working formula.
  • PLDI is seen by external (to PL) researchers as perhaps the primary PL venue for papers that sit at the intersection of PL and other areas (e.g., networks, security, etc). Having submissions from those communities allows for a healthy interaction between PL and other areas. Merging PLDI into PACMPL together with conferences which are not seen the same way externally, combined with the very different format of PACMPL (e.g., an unusual one column format) vs. the format of conferences which are currently coordinated with PLDI (e.g., ASPLOS) may deter those external researchers from submitting to PLDI (e.g., PLDI deadline is tightly coordinated with ASPLOS notification), thus potentially affecting impact and relevance of PL research overall.
  • PLDI need not commit straight away. PLDI could wait for a year, or three, to see how PACMPL works out—in particular, to check that PACMPL is indeed included in journal indices.
  • Joining PACMPL is effectively irrevocable; no one knows what the impact will be on promotion and tenure processes of merging multiple conferences together with PLDI into a single journal. (In particular, according to the ACM DL, PLDI averages 32.95 citations/paper. In contrast, POPL averages 26.74 and ICFP averages 8.79. OOPSLA's figures are broken, because the research papers have been mixed up with posters etc.) Keeping PLDI separate eliminates this possible confusion and maintains the well-established PLDI brand.
  • The PACMPL proposal does not solve an actual problem faced by PLDI, which is successful: it is currently one of the most prestigious and selective PL conferences. It is not clear that joining PACMPL will help either people who already publish in PLDI or those who theoretically do not due to its non-journal status.

The survey

Here again is a link to the survey.

Comments

  1. I note that the reasons against don't even bother to mention Open Access.

    ReplyDelete
    Replies
    1. SIGPLAN is currently subsidizing Open Access fees for PACMPL; in principle, SIGPLAN could do the same for PLDI, since it's the same number of papers either way.

      In any event, the ACM recently changed its terms for paper posting by authors, so the ACM paywall is no longer much of an issue. Authors can now freely post accepted versions of their papers to arxiv.org, their websites, etc., in addition to the authorizer links.

      (From author.acm.org/main.html)

      Authors can post the accepted, peer-reviewed version prepared by the author-known as the "pre-print"-to the following sites, with a DOI pointer to the Definitive Version of Record in the ACM Digital Library.

      On Author's own Home Page and
      On Author's Institutional Repository and
      In any repository legally mandated by the agency funding the research on which the work is based and
      On any non-commercial repository or aggregation that does not duplicate ACM tables of contents, i.e., whose patterns of links do not substantially duplicate an ACM-copyrighted volume or issue. Non-commercial repositories are here understood as repositories owned by non-profit organizations that do not charge a fee for accessing deposited articles and that do not sell advertising or otherwise profit from serving articles.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. I am, personally, persuaded by the argument that publications are effectively open access now, without paying for them to be Gold OA, thanks to the points Emery quotes and thanks to OpenTOC (see http://www.sigplan.org/OpenTOC/). But I ran a survey and 300+ people disagreed with me 2-to-1, so I relented and PACMPL is Gold OA. SIGPLAN won't subsidize forever, but I expect fees will go down, and payment by registration subsidy will be very small.

      Delete
    4. As my student Michael Coblenz recently pointed out in a discussion at ICSE, OpenTOC is visible to people who know about ACM's policies--but most of those are members or institutional subscribers and have access through the paywall anyway. Open Access is primarily about reaching the people who only occasionally interact with ACM conferences. Those people don't know about OpenTOC and will be frustrated by going to the canonical source and seeing it paywalled.

      If we want open access, we should go for open access. Gold OA is the only real OA: making the canonical source freely available.

      Delete
    5. We should definitely publicize OpenTOC more. (Also, I have yet to Google for a paper from a SIGPLAN conference without finding a freely available PDF version. YMMV, but I do not see this as a practical concern any longer.)

      In any event, here's the OpenTOC for SIGPLAN conferences.

      http://www.sigplan.org/OpenTOC/

      Delete
    6. I just started Googling the PLDI 2016 program. The second paper listed ("A Distributed OpenCL Framework using Redundant Computation and Data Replication") had no free PDF version on the first page of Google hits. Maybe a majority of papers are available, but I frequently run into those that aren't. And it's a major pain even for me whenever I'm not on my university network (at the very least I have interrupt my flow to set up VPN).

      Anyway, I agree about publicizing OpenTOC. I have a modest proposal: on the page for each SIGPLAN paper in the ACM Digital Library, create a link labeled "freely available version" to the OpenTOC page for that paper. This advertising approach would reach exactly the target audience: the people who go to the digital library and get blocked by the paywall. Bonus: ACM pays nothing for this form of advertising!

      If you really believe that OpenTOC is effectively open access, there should be no problem with the advertising strategy above.

      Delete
    7. Using scholar.google.com, the second and third links (from "all versions") have PDF links. :)

      https://scholar.google.com/scholar?cluster=5856863462739188162&hl=en&as_sdt=0,22

      But Yes, great idea about OpenTOC. We should ask ACM to do this.

      Delete
  2. I note none of the arguments for or against mention how publication venues relate to doing better science.

    One advantage of a journal over conferences (it seems) is the possibility of longer papers. I will mention only one of the reasons to want to do that. In the past in the PL world, the technical material in published research papers was relatively short. There was not much prior work to discuss, not much about the experimental or theoretical setup, the results were simple, and the practicalities straightforward. These days all of these have gotten much, much bigger, but page limits on conferences have remained the same. This means authors are trying to do as much presentation compression as they can. Guy Steele has argued recently that notation in POPL papers have become compressed enough to be almost unintelligible to many, and authors spend no space explaining their compressions (their new, compressed notation). Taking expansive material and compressing it in English is a task trained writers find difficult. PL authors are not trained writers. The result is that papers are becoming more and more incomprehensible based solely on page limits.

    I argue that if papers could be more comprehensible, more and better research could be done because more people can understand the problem area and understand it better.

    ReplyDelete
    Replies
    1. Page limits are an orthogonal concern - just as an example, OOPSLA has had 20 pages as its limit (in the old sigplan format) for a long time.

      Delete
    2. Paper length is orthogonal to journal versus conference.

      Paper submission length is a function of reviewer exhaustion. Most journals in other sciences have page limits for the same reasons we do.

      The theory is that a conference reviewer has 20 +-15 papers to review, so that limits paper length, whereas a journal reviewer has 1. This theory does not follow our community. We need the established active researchers (not students) in our field to review at least ~8 to 15 papers a year to continue to innovate and publish at current levels.

      Here's how I estimate that number. In 2012, SIGPLAN co-sponsored around 700+ conference papers. With acceptance rates between 15 and 40%, let's assume 30% on average, that is 2100 submissions with ~3.5 reviews a peice (most have 3 many have 5), that yields 7350 reviews. If there were 1000 active PL reviewers, 8 reviews per year or with 500, 16 reviews per year. HOTCRP could tell us the active number of reviewers for a more precise estimate.

      Delete
  3. "If PLDI were to join PACMPL from 2018, then all four of the top SIGPLAN conferences would be included from the same point in time (namely, everything after July 2017)."

    In situations were the outcomes aren't clear, putting all your eggs in one basket is probably not the best idea.

    "If PLDI waits to join, PLDI authors and readers will miss out on Gold OA in the meantime."

    That seems like a small price to pay in order to avoid a situation in which all top PL venues are locked in a scheme that turns out to be a bad idea. Such a situation is very unlikely in my opinion, but possible. (In terms of eggs, using multiple baskets almost guarantees that some eggs will be lost. But, using multiple baskets is still a good idea because it also guarantees that some eggs will not be lost.)

    So, I think that PACMPL is a good idea. But, I'm not 100% sure. For that reason, I'd be happy for PLDI to wait for a while.

    ReplyDelete
  4. There seems to be the premise that PACM will get impact factor and the number will be high. It is not clear this will happen, but anyway let's assume it will achieve this goal. Now, even if for us internally POPL issues are different from PLDI issues, and from ICFP issues, for the number they will be the same. Obviously this will essentially mean that if I am in a country where journal bean counting matter, I'd better go most of the time for the easiest of these (and some are orders of magnitude easier for the same reward). This journal thing may not affect anybody in US, so one effect of the measure may only be that it will split the community into conferences for countries XYZ and conferences for countries UVW (in terms of where majority of the effort goes, of course everyone can visit the any conference).

    ReplyDelete
  5. I'm surprised that one "con" was omitted: having quite different paper formats for the major SIGPLAN conferences means that it's relatively time-intensive to reformat a paper to submit to one of them, when the last revision was rejected from another! (There are quite a few of us now who write papers that we consider sending to any of the 4 main SIGPLAN conferences, based on the deadline schedule.)

    ReplyDelete
    Replies
    1. Er, I meant "con" of keeping the status quo.

      Delete
    2. At the very least, there's now a common latex template where you just change an option to switch format. Of course you'll have to fix line breaks, but on resubmission you probably have to anyway due to revisions.
      Double-column figures, OTOH, are indeed especially painful.

      Delete
  6. PLDI, POPL, OOPSLA, and ICFP are already indexed through SIGPLAN Notices. If PACMPL is indexed, it will be indexed as a single journal. (If not, then why the indexing services would not index the conferences is beyond me.)

    Does merging all our conferences for impact ratings have any benefits for our community? Maybe it reduces the pressure to publish in PLDI and POPL vs. OOPSLA and ICFP? (Note that Emery's csranking.org currently includes only PLDI and POPL for various reasons, see the FAQ, but the OOPSLA measurements in dblp he uses are wrong due to inadvertent inclusion of tutorials, posters, etc. as OOPSLA papers, but I digress.) Is smushing a lot of PL into one blob good or bad?

    I think it really sends a message that we don't value at all research that appears in the other venues such as ISMM, VEE, CGO, PACT, PPoPP, etc. especially in the short run. Lots of great work that has highly influenced PL research appears in these other venues. Do we want to make the incentives even more lopsided than they already are? Will researchers give up on even trying to publish in other venues? We could really hurt them and hurt the PACMPL venues by further increasing their submissions.

    In our current system, PLDI has published an amazing amount of high quality research, e.g., 80% of PLDI papers are still being cited 10 years after publication see: http://cm.cecs.anu.edu.au/citation/PLDI/

    PLDI differs in this regard (and many others) from POPL, OOPSLA, and ICFP. Do we as a community not value these differences? I think keeping the individual character and prestige of the particular conferences is to the benefit of the PL community.

    The prestige and citation rates of venues matter (even if you can game it). For hiring PhDs and making tenure decisions, papers do not have time to get cited. Using venue as a standin is a reasonable thing to weigh in those decisions because it represents the opinions of three to five PL experts on the PLDI committee for this new work.

    PACMPL is an experiment. I urge the community to exhibit patience. PLDI can wait until the first adopters prove it a good or bad idea to make its decision.

    ReplyDelete
  7. Putting the arguments against joining aside for a second, the arguments in favor of joining PACM can be broadly classified as (number of bullet point in brackets):
    * Administrative benefit [1] - which assumes that PACM can and will become a stronger “brand” than PLDI.
    * Popular vote [2,3] - where [2] is about *a* journal, and [3] is not that conclusive if you look at community "veterans" (>=2 PLDI papers). Regardless, a vote is more meaningful now that we have a better understanding of pros/cons.
    * Open access [4,7] - which I view (maybe incorrectly) as orthogonal.
    * Risk mitigation [5,8] - why joining isn’t fatal. These are not really arguments in favor.
    * Nice to have [6].

    So it all boils down to the question of whether PACM will become a stronger “brand” than PLDI. For that, we need to better understand how will the impact factor of PACM be determined. Does anyone have data that can be used to compute the expected impact factor of PACM (with/without PLDI)? Based on this data, can we expect PACM to really become a stronger brand?

    ReplyDelete
  8. If the "brand" argument is based on "quality" as measured by citation rates and citation survival rates, PLDI is the best of the four. Assuming the trends stay the same, PACMPL's metrics benefit by including PLDI and PLDI's degrades.

    http://cm.cecs.anu.edu.au/citation/PLDI/
    http://cm.cecs.anu.edu.au/citation/POPL/
    http://cm.cecs.anu.edu.au/citation/ICFP/
    http://cm.cecs.anu.edu.au/citation/OOPSLA/ (Because of ACM DL errors, OOPSLA includes posters, etc., diluting both metrics substantially.)

    ReplyDelete
  9. People, a common journal is not solving any problems. PLDI is well-recognized and needs no new branding. I understand the desire to boost OOPSLA and ICFP, but I see no argument that this will in fact help. It would be great if somebody actually argues more in-depth about that.

    On the other hand, the PL community needs more unity. Much better approach, in my opinion, would be a common SIGPLAN issue/magazine dedicated to popularizing PL research. And by popularizing, I mean not only among researchers but also among programmers in general.

    See, for example, the magazine published by the CHI people:

    http://interactions.acm.org/about

    or the popular series by the AMS fellows:

    http://www.ams.org/samplings/math-history/happening-series

    News "from the kitchen" presented in an easygoing and engaging style will spark much more curiosity, and help PL people get interested in what other PL people are doing.

    (Btw, the counterpoint about the impact factor nonsense is also very strong. The PL community successfully avoided it for a long time. Do we want to enter it now?)

    ReplyDelete
  10. > PLDI is well-recognized and needs no new branding.

    It is well recognized within PL community. Note that there will be situations where a researcher will have his/her work assessed by people who do not specialize in PL (yes, that happens!). And, as noted in the main text there are areas (Eastern Europe here!) where conference publications are not being properly recognized.

    That being said, turning PLDI publications into journal ones may not offer much help for people from such areas. The reason is that another expectation is that a researcher publishes in *different* journals. Publishing all the papers in the same journal over and over is sometimes considered an indication that one has a "good relationship" with the editor.

    Also, doesn't SIGPLAN Notcies already play the role PACMPL is intended to fulfil?

    ReplyDelete
  11. My experience is that outside academic institutions or research labs, PLDI is not only a well-recognized brand for PL research publications, it is often the only such brand. It is THE place many people look for new results on compilers or runtime systems. I would much rather not have us sacrifice that to support fundamentally flawed tenure evaluation processes.

    ReplyDelete
  12. I'm rather mystified by the concern about branding. PLDI is PLDI. There is no proposal to change it in any fundamental way whatsoever. This is essentially an alpha-conversion. If you prefer to keep citing your PLDI papers on your CV as a "conference paper" published in "PLDI 2018" rather than in "PACMPL (PLDI) 2018" or whatever, no one would stop you. And for people outside academic institutions and research labs, why would they stop looking at papers published in PLDI, just because the front page of the proceedings will say that it was published as a PLDI issue of PACMPL? I feel I must be missing something, because this seems like a total non-issue.

    Put another way, no one currently confuses PLDI with POPL, ICFP, or OOPSLA, despite the fact that they are all published as issues of SIGPLAN Notices. Why would changing "SIGPLAN Notices" to "PACMPL" suddenly cause confusion?

    ReplyDelete
    Replies
    1. SIGPLAN Notices is a newsletter that republishes conference papers. If PACMPL intends to be considered as a real journal, then it doesn't seem to be in the same category as SIGPLAN Notices, and it would seem to be unclear that putting a PACMPL paper in the conference section of a CV would be the right thing to do.

      Delete
    2. I don't see why anyone would seriously object to seeing a PLDI (or POPL or ICFP or OOPSLA) paper in the "conference paper" section of someone's CV, even after the switch to PACMPL. The purpose of PACMPL was to help people who are trying to make the case to their organizations that our conference publications are at least at the level of journal publications in other fields -- by literally turning our conference pubs into journal pubs. For people who do not need to make such a case (including me), they should be totally unaffected by the move to PACMPL, and should be able to keep on with business as usual. (This was certainly how the idea of PACMPL was sold to me when I was on the SIGPLAN EC.)

      Delete
    3. FWIW the original proposal was one journal per conference (as in, "PACM PLDI", etc.). That is not the current proposal.

      Delete
    4. This comment has been removed by the author.

      Delete
    5. I still don't see the branding issue. The branding issue is: Will PACMPL "dilute the brand" of PLDI? And the answer is no, because the only way to dilute the brand of PLDI is for papers to be published under the brand "PLDI" that were not published at PLDI. That is not going to happen. There will be PACMPL papers published in PLDI issues and those published in issues for other conferences. But a PLDI paper will remain a PLDI paper. So what again is the branding issue?

      Delete
    6. Emery, there was never a proposal for one PACM journal per SIGPLAN conference - or even for a PACM journal devoted to a single SIGPLAN conference. The only proposal was for PACM PL, which OOPSLA started and other conferences have joined.

      Delete
    7. This may depend on what you mean by _proposal_. Mike sent a mail to everyone on the PLDI Steering Committee detailing that that was indeed the original plan (on 2/11/2016). The mail makes it quite clear that the intent would be a different journal for each conference (e.g., PACM PLDI and PACM OOPSLA, and by extension, PACM POPL, PACM ICFP, etc.) rather than all being merged into one, but Mike indicated this was rebuffed by the ACM Publications Board

      Delete
    8. That’s not my reading of Mike’s message, Emery. SIGPLAN EC discussed a one-journal-per-conference proposal, but this was explicitly vetoed by Pubs Board. The formal proposal that Mike sent to PLDI-SC on 5th Feb 2016 (to which the message on 11th was a follow-up) was clearly about a proposal for a single journal, PACMPL, consisting of OOPSLA and PLDI in the first instance. PLDI ultimately declined to join, so the proposal went ahead with OOPSLA only. ICFP and POPL joined later.

      Delete
    9. I'm still not sure why, as far as branding is concerned, having a separate PACM journal for each conference is any different from having a single PACMPL journal with clearly separately named issues for different conferences. Either way, the name PLDI would appear in the citation of any PLDI paper and would not appear in the citation of any non-PLDI paper.

      Delete
    10. The reason including PLDI in PACMPL or not makes a BIG difference to our community is because the PLDI name brand is supported and "proven" by bibliometrics computed by DBLP, Google Scholar, Microsoft Academics, etc. If PLDI joins PACMPL it becomes TOTALLY OUT OF OUR CONTROL, if these analysis tools compute metrics only for PACMPL, or for each of the venues it publishes, or both. The POPL, OOPSLA, and ICFP Steering committees made their decisions to take on this risk without an open community discussion. The PLDI SC voted twice to not join in at this time. Activist SIGPLAN EC member(s) feel the community should speak about this issue. So here we are.

      Delete
    11. Kathryn, why are you worried that DBLP etc will change their behaviour if the PLDI proceedings switch from being in SPN to being in PACMPL? There will still be a PLDI conference. The existing indices have no problem with HIPEAC, ICLP, etc, which also appear in journals mixed with other papers. They are not under our control at the moment. (I don't believe that DBLP computes any metrics or looks in any way at citations.)

      And I think that you're misrepresenting the PLDI SC votes. The SC has indeed voted twice, and has twice been pretty evenly split, for and against - hence this broader discussion. "Not voting to join" is by no means the same thing as "voting not to join".

      Delete
    12. In response to Jeremy - I received permission from Mike Hicks to post the following quote from his mail, which I think is pretty clear.

      -----
      When we were discussing this proposal among the SIGPLAN EC, the question of branding was very important. We considered two possible approaches:

      1) Each conference has its own PACM journal, e.g., PACM PLDI and PACM OOPSLA. This makes branding crystal clear.

      2) We ran the above idea past XXX, of the ACM pubs board, and it seemed likely that this approach would not fly. [...]
      -----

      Delete
    13. Right. That idea was discussed by SIGPLAN EC, but rejected by Pubs Board as a non-starter before it ever came to PLDI SC, let alone becoming a formal proposal.

      Delete
    14. > Kathryn, why are you worried that DBLP etc will change their behaviour if the PLDI proceedings switch from being in SPN to being in PACMPL? There will still be a PLDI conference. The existing indices have no problem with HIPEAC, ICLP, etc, which also appear in journals mixed with other papers.

      It's less clear the *existing* branding is in danger. But for those who ignore PLDI because it's no journal, impact factors are likely to also matter. And those are usually per-journal—I understand that when Nature split into multiple journals its impact factor was affected. I'd guess PLDI won't get better treatment than Nature.

      Delete
    15. Certainly, the impact factor will be for PACMPL as a whole, and likely lower than it would be were PLDI to have the journal to itself. But it'll be higher than it currently is, because currently PLDI is in SIGPLAN Notices.

      Delete
    16. This comment has been removed by the author.

      Delete
    17. Regarding impact factors - merging PLDI into a single publication with POPL, OOPSLA, and ICFP would likely reduce PLDI's impact as measured by Google Scholar (assuming that Google Scholar will not make a special case for PACMPL volumes).

      By h5-index, POPL is 7% higher (48 for POPL vs. 45 for PLDI), OOPSLA is 20% lower (36), and ICFP is 38% lower (28).

      https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=eng_softwaresystems
      https://scholar.google.com/citations?hl=en&view_op=search_venues&vq=ICFP

      Delete
    18. I would not expect Google Scholar to make a special case for PACMPL. But do you think they are likely to stop measuring PLDI if it were to switch from SPN to PACMPL?

      Delete
    19. 1. State-mandated bibliometrics won't look at Scholar, more likely at Clarivate.
      2. Scholar claims to follow publisher-declared metadata, annotated via meta tags on webpages (or heuristics). Details below.

      FWIW, VLDB also created a journal (Proceedings of VLDB), but Scholar kept indexing VLDB:
      https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=eng_databasesinformationsystems

      Regarding docs:
      From Emery's Scholar indexes, I got to
      https://scholar.google.com/intl/en/scholar/metrics.html#inclusion and finally to webmaster docs on https://scholar.google.com/intl/en/scholar/inclusion.html#indexing.

      Each article page declares its venue. For articles declaring to be both a conference and a journal paper... Scholar doesn't document its behavior. In a sample PLDI paper webpage (http://dl.acm.org/citation.cfm?id=2908108), ACM DL adds such meta tags for both SPN and PLDI. I found no docs on this situation. Scholar claims to prefer one article per webpage.

      Delete
  13. I'm not at all concerned with how things are cited in a CV. I think it's our job to facilitate progress in the field. We shouldn't get in the way of assessing individual impact, but that's fundamentally the problem of people doing hiring and promotion. If they think they can reasonably continue to do that indefinitely by looking at just traditional bibliometrics, they're hopelessly wrong, and they will continue to be hopelessly wrong no matter what we do.

    But when I see a reference in a paper, I would like to be able to tell whether it was published in PLDI, and thus reviewed by people who, if nothing else, have biases similar to mine. And I expect many people are more likely to follow up on the reference if it says PLDI. We already have a problem with PLDI papers that are cited as SIGPLAN Notices, which we should try to address; I don't think we should confuse matters further by introducing another alias.

    ReplyDelete
    Replies
    1. PACMPL will only help with that problem. At the moment, PLDI papers appear both in the proceedings and in SPN, and so have two different DOIs. With PACMPL, there would be just one DOI, and just one citation to grab from any index.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. This is an important issue, but it's a red herring in the context of the PACM debate.

      Cited correctly, the SPN version is clearly PLDI. There's no confusion (see the DL for details, e.g. http://dl.acm.org/citation.cfm?id=2737965). Same goes for PACM PL. In the former, PLDI is the 'series', in the latter, PLDI is the 'issue'.

      However, our community has a systemic problem of poor citation practices, not helped by the DL providing very poor bibtex. In both cases, the PLDI may be dropped by those not being careful in their citation.

      If one really wanted to push this argument (I don't), you could argue that PACM will only make things worse. Right now most people cite PLDI as the conference (where PLDI can't be avoided in the citation), while a few use the SPN style which is open to loss of affinity to PLDI (if series is dropped). With PACM PL everyone will have to use the form that is open to misuse and loss of affinity to PLDI (if issue is dropped).

      Delete
    4. I don't follow, Steve. The DL provides two different ways of citing a PLDI paper, one as a "journal article" in SPN and one as a "conference paper" in the PLDI proceedings. The SPN one doesn't mention PLDI, whether in BibTeX format, or EndNote, or ACMRef. For example, for the paper you cite, the ACMRef recommendation is:

      Nuno P. Lopes, David Menendez, Santosh Nagarakatte, and John Regehr. 2015. Provably correct peephole optimizations with alive. SIGPLAN Not. 50, 6 (June 2015), 22-32. DOI: http://dx.doi.org/10.1145/2813885.2737965

      which does not mention PLDI. I don't know about EndNote, but in BibTeX the "article" entry type (in standard bib styles) ignores any "series" field. So even if you were to add that metadata in your .bib file it would not make it through to the generated .bbl output.

      In contrast, with PACMPL there will be just one DL entry for a PLDI paper, not two, and the PLDI name will be preserved in the "issue number" field. Someone who simple-mindedly scrapes the citation data from the DL cannot help but do the right thing.

      Delete
    5. Yes. Please re-read what I wrote.

      There are at least two fundamental problems:

      1. The DL provides bad bibtex systematically. Take a look at the example I gave you. Look at the DL's own rendering of the citation (lower left): "ACM SIGPLAN Notices - PLDI '15, Volume 50 Issue 6, June 2015, Pages 22-32". Yes, I know their bibtex does not reflect this.
      2. Authors are systemically sloppy in their citation form. Consequently, issue numbers are frequently dropped from citations. If this happens to PACM PL, we lose affinity with PLDI.

      Delete
    6. Oh, I see what you mean. The only way to get that effect in BibTeX would be to put "PLDI" in the name of the journal. That's pretty illogical, and I have never seen it done.

      I infer that people who cite PLDI papers as being in SPN always do so simply by grabbing the first BibTeX entry from the DL (it's always "SIGPLAN Not.", never "Notices"); such people will presumably not actively break a PACMPL entry.

      Delete
  14. My perception of PACMPL has moved from being generally supportive to "wait and see". Let me explain.

    First of all, let me clearly state my motivation for SIGPLAN and computing systems conferences to move boldly towards a combination of a conference and journal publication model. The journal first model (TACO/HiPEAC, PVLDB, SIGGRAPH) set the way, but we can improve it with the ability to discuss the papers in a PC (probably not physical, and this will definitely not work with tools like manuscript central or any journal management system I know of, fortunately we have HotCRP). At this point, I believe PACMPL is addressing the wrong problem. While being strongly in favor of a profound change in the publication model, and definitely in favor of making PLDI proceedings a journal, my position has changed over the last few months. I am now inclined to wait and see, and have further discussions with ACM to make PACMPL a better journal. Let me explain.

    One important concern with PACMPL that is not often voiced is that conferences that do not get in may lose in visibility and impact. PPoPP is currently my main concern. But CGO is also on the list. Both conferences regularly get top papers (PPoPP even more so considering its very low acceptance rate) that would in theory fit PLDI or POPL, but they do not get in because they are not "exciting" enough, in one way or another. Ben Zorn once had an interesting graph showing how compiler optimization/performance papers moved away from PLDI in the early 2000s (alongside the emergence of GCO). Note that moving to 2 deadlines per year may bring some of these papers back to PLDI (or POPL). People publish in a variety of conferences. Pretty much everyone, but particularly those with more papers in specialized/focused conferences will be negatively impacted by the move to PACM. Unless PACM is made much more inclusive of course. The main reason I can think of for not immediately making PACMPL more inclusive is that this would lower the global impact factor of the journal. But (1) if someone would really care, ACM can still be armtwisted to create gazillions of journals rather than just one, or the parasitic bibliometrics businesses could be made obsolete or forced to count differently, the world is changing fast anyway... and (2) I could not care less. In fact, the first argument of the people against (see the blog introduction) is that we should not put bibliometrics as a motivation for doing anything. So let's be consistent and discard the bibliometrics argument altogether when discussing which conference gets into PACMPL and which does not.

    Gold Open Access has always been my main concern and the act it is tied to the move to PACMPL, that ACM does not offer Gold OA for proceedings, is an artificial constraint and complication. I will remain worried by the move to Gold OA as long as ACM does not lower the fees even further, with a fully transparent fee structure, or at the very least that all past proceedings get published under more relaxed terms alongside with the move of new volumes to OA (hopefully the same license, if the contracts with authors allow that). There are so many Green OA alternatives anyway, Gold OA is seen as a luxury by many, and non-OA as a thing of the past anyway.

    In the end, I voted against the move to PACMPL. We should wait and see and urge ACM to make bolder steps. I am in favor of making proposals through SIGPLAN, based on a careful study of the ongoing journal first publications. I know that small steps are generally more pragmatic, but in this situation I am not sure PACMPL is a small step in the right direction. I am also confident that ACM will soon face so much pressure on all sides that it will have to make its conference business evolve much more dramatically, and SIGPLAN should be ready to propose a bolder, practical solution for when that time comes.

    ReplyDelete
  15. The decision of the PLDI steering committee is here

    ReplyDelete

Post a Comment